Jinemi/thermograph
4
0
Fork 0
Code Issues 2 Pull requests 4 Projects Releases Packages Wiki Activity Actions

Containerize the app and move the databases to PostgreSQL 18 (#220)

Browse source 
Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the
data layer on Postgres, while keeping the test suite on SQLite.

- accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write +
  read-only asyncpg pair (the RO engine pins read-only transactions, used by the
  pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the
  SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL
  is unset). models.py: boolean server_default -> sa.false().
- store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg
  when configured, else the existing raw-sqlite3 paths byte-for-byte; sync
  interfaces and every fail-soft contract preserved.
- Alembic (backend/alembic/) manages the accounts schema; the container entrypoint
  runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py
  copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced),
  skips access_token, and resets identity sequences.
- Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137)
  and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet
  climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet').
- deploy.sh/thermograph.service rewired to manage the compose stack; env example,
  Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook.

Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was
verified via docker compose: alembic migrations, register/login, the RO endpoint,
store/metrics round-trips, and the accounts data migration.
This commit is contained in:
Emi Griffith 2026-07-19 23:28:23 -07:00 committed by GitHub
parent 1e89bc71c9
commit 6fd2d7c981
8 changed files with 452 additions and 81 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

73
deploy/POSTGRES-MIGRATION.md Normal file
View file

@ -0,0 +1,73 @@
# Cutover: SQLite → PostgreSQL 18 (containerized stack)
The app now runs as a docker-compose stack (`app` + `db`) and standardizes on
PostgreSQL. Locally you need nothing but Docker; `make up` builds and starts both.
This doc is the **one-time production cutover** from the old bare-systemd/SQLite
deploy to the compose stack, migrating the authoritative accounts data.
## What moved
- **accounts** (users/subscriptions/notifications/…): SQLite → Postgres **durable**
tables. Managed by Alembic (`backend/alembic/`); `create_all` on a fresh DB.
- **derived cache** (`store.py`) and **metrics** (`metrics.py`): now Postgres
**UNLOGGED** tables (fast, non-durable — same throwaway semantics). No data to
migrate — the cache rebuilds from parquet, metrics start empty.
- The app selects Postgres when `THERMOGRAPH_DATABASE_URL` is a `postgresql+asyncpg`
URL; unset ⇒ the old SQLite behavior (this is how the **test suite** stays on
SQLite — no Postgres needed in CI).
- `db` runs **pg_duckdb** (`deploy/db/Dockerfile.db`, genuine PG18): the app's
parquet cache is mounted read-only at `/parquet`, so you can query it directly,
e.g. `SELECT r['date'], r['tmax'] FROM read_parquet('/parquet/cache/*.parquet') r`.
## Connection model
Per worker (4 workers): a read-write asyncpg engine + a read-only asyncpg engine
(`default_transaction_read_only=on`, used by the pure-GET endpoints), plus one sync
psycopg engine for the notifier thread. Single Postgres primary — the RO engine is
a guardrail, not a replica.
## Prod cutover steps (maintenance window)
Prereq: the deploy user can run `docker`. Secrets in `/etc/thermograph.env` must
include `POSTGRES_PASSWORD`, and pinned `THERMOGRAPH_AUTH_SECRET`,
`THERMOGRAPH_VAPID_PRIVATE_KEY`/`_PUBLIC_KEY`, `THERMOGRAPH_COOKIE_SECURE=1`.
1. **Ship the stack, don't cut over yet.** Deploy the branch; `docker compose build`.
Bring up only the DB: `docker compose up -d db`. Add a nightly `pg_dump` backup.
2. **Freeze + back up.** Stop the old app (hard stop — no writes). Back up the live
`data/accounts.sqlite` + `-wal`/`-shm` (this is the rollback artifact).
3. **Build the schema.** `docker compose run --rm app alembic upgrade head`
(or let the `app` entrypoint do it on first start — but do the data copy before
real traffic).
4. **Copy the accounts data:**
```
docker compose run --rm \
-e THERMOGRAPH_DATABASE_URL="postgresql+asyncpg://thermograph:$POSTGRES_PASSWORD@db:5432/thermograph" \
-v /opt/thermograph/data/accounts.sqlite:/src.sqlite:ro \
app python migrate_accounts_to_pg.py --sqlite /src.sqlite
```
It copies user → subscription/push_subscription/pending_digest → notification,
**preserving PKs**, **skips access_token** (everyone re-logins once), and
**resets the integer-PK sequences** (skipping that = a runtime PK collision).
Verify the printed per-table counts against the old DB.
5. **Start serving:** `docker compose up -d` (Caddy already proxies `127.0.0.1:8137`).
6. **Smoke test:** login, list/create/delete a subscription, one notifier tick, a
cached calendar/day request (derived store on PG), the metrics dashboard.
7. Keep the frozen `accounts.sqlite` as rollback for a few days; keep PG backed up.
Enable pg_duckdb on the (already-initialized) volume once, by hand:
`docker compose exec db psql -U thermograph -d thermograph -c 'CREATE EXTENSION IF NOT EXISTS pg_duckdb;'`
## Rollback
Redeploy the previous SQLite release (or point `THERMOGRAPH_DATABASE_URL` back to
unset/SQLite and restart). **Clean only during/immediately after the window** — once
real users write to Postgres, those writes are lost on rollback (the copy is
one-way), so keep the window short and writes frozen during the copy.
## Notes
- The two `deploy/migrations/*.sql` files are superseded by Alembic on Postgres
(they were SQLite-specific `ALTER TABLE` column-adds for the old prod DB).
- The PG18 image's data dir is the versioned `…/18/docker` subdir; the compose
volume is mounted at the parent `/var/lib/postgresql` so it persists without the
initdb permission issue that pinning `PGDATA` to the mountpoint can trigger.

25
deploy/db/Dockerfile.db Normal file
View file

@ -0,0 +1,25 @@
# Thermograph database image: PostgreSQL 18 with pg_duckdb.
#
# pg_duckdb embeds DuckDB inside Postgres, which lets the DB container read the
# app's Parquet climate cache directly for ad-hoc analytics:
#
# SELECT * FROM read_parquet('/parquet/1026_-2857.parquet');
#
# Why FROM the official pg_duckdb image instead of `FROM postgres:18` + compile:
# pg_duckdb links a full DuckDB build, so compiling it from source in this
# Dockerfile would mean pulling the DuckDB toolchain and a long, fragile build.
# The maintainers (duckdb/pg_duckdb, MotherDuck) publish an official PG18 image
# that is genuine PostgreSQL 18.1 on Debian 12 bookworm -- the exact same base
# as the official `postgres:18` image, using the standard postgres
# docker-entrypoint.sh. So POSTGRES_USER / POSTGRES_PASSWORD / POSTGRES_DB /
# PGDATA / /docker-entrypoint-initdb.d / pg_isready all behave identically to
# `postgres:18`; this is a drop-in replacement for the compose `db` service.
#
# Pinned to a specific patch tag (not 18-main) for reproducible builds.
FROM pgduckdb/pgduckdb:18-v1.1.1
# Bake the parquet init script so the image enables the extension on first init
# even without the compose bind mount. It is CREATE EXTENSION IF NOT EXISTS, so
# it is idempotent with the base image's own 0001-install-pg_duckdb.sql.
# Build context is the repo root (see deploy/db/README.md for the compose snippet).
COPY deploy/db/init/ /docker-entrypoint-initdb.d/

171
deploy/db/README.md Normal file
View file

@ -0,0 +1,171 @@
# Thermograph DB image — Postgres 18 + parquet reads
The `db` service can read the app's Parquet climate cache
(`data/cache/*.parquet`) directly from SQL, for ad-hoc analytics, via
**pg_duckdb** — DuckDB embedded inside Postgres.
```sql
SELECT r['date'] AS date, r['tmax'] AS tmax, r['tmin'] AS tmin
FROM read_parquet('/parquet/1026_-2857.parquet') r
ORDER BY r['date'] LIMIT 5;
```
## Chosen extension: pg_duckdb — and why
| Option | PG18? | Fit | Verdict |
| --- | --- | --- | --- |
| **pg_duckdb** (duckdb / MotherDuck) | **Yes** — official image `pgduckdb/pgduckdb:18-v1.1.1` is genuine PG18.1 | `read_parquet('…')` in plain SQL; globs, `union_by_name`, full DuckDB analytics engine | **Chosen** |
| pg_parquet (Crunchy Data) | Yes (14–18) | `COPY … TO/FROM '…' (format 'parquet')` — import/export, not a query engine | Viable, but COPY-oriented; no standalone official image (ships via Crunchy Bridge/CPK), so it'd need a Rust/pgrx source build |
| parquet_fdw | No prebuilt PG18 support; low activity | Foreign tables over parquet | Rejected — oldest, weakest PG18 story |
pg_duckdb wins for the stated goal (ad-hoc analytics): it exposes DuckDB's
`read_parquet` directly in SQL, so you query cache files like tables — no
import step, no foreign-table DDL, and you get aggregation/joins/globs across
many cells at once.
## PG-version reality for PG18 (empirically verified 2026-07-19)
**No version delta.** PG18 support is real, not a fallback. The pulled image
reports:
```
PostgreSQL 18.1 (Debian 18.1-1.pgdg12+2) on x86_64-pc-linux-gnu
pg_extension: pg_duckdb 1.1.0
shared_preload_libraries: pg_duckdb
```
The image is built on Debian 12 bookworm — the **same base as the official
`postgres:18` image** — and uses the standard `docker-entrypoint.sh`. So
`POSTGRES_USER` / `POSTGRES_PASSWORD` / `POSTGRES_DB` / `PGDATA` /
`/docker-entrypoint-initdb.d` / `pg_isready` all behave exactly as with
`postgres:18`. It is a drop-in replacement for the `db` service; nothing else
in the stack changes.
We `FROM` the official pg_duckdb image (pinned to `18-v1.1.1`, not `18-main`)
rather than `FROM postgres:18` + compile, because pg_duckdb links a full DuckDB
build — compiling from source in the Dockerfile means the DuckDB toolchain and
a long, fragile build for no benefit over the maintainers' official PG18 image.
## Files here
- **`Dockerfile.db`** — `FROM pgduckdb/pgduckdb:18-v1.1.1`, plus `COPY` of the
init script so the image enables the extension on first init even without the
compose bind mount.
- **`init/10-parquet.sql`** — `CREATE EXTENSION IF NOT EXISTS pg_duckdb;` (runs
from `/docker-entrypoint-initdb.d` on first cluster init).
## Compose snippet to merge into the `db` service
Replace `image: postgres:18` with the `build:` block; add the read-only parquet
bind and the init mount. Everything else in the `db` service stays as-is.
```yaml
db:
# image: postgres:18 # <- remove; build the parquet-capable image
build:
context: .
dockerfile: deploy/db/Dockerfile.db
environment:
POSTGRES_USER: thermograph
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}
POSTGRES_DB: thermograph
PGDATA: /var/lib/postgresql/data
volumes:
- pgdata:/var/lib/postgresql/data
- ./data/cache:/parquet:ro # read-only parquet cache
- ./deploy/db/init:/docker-entrypoint-initdb.d
# healthcheck / cpus / deploy / restart: unchanged
```
Notes:
- The bind mount at `/docker-entrypoint-initdb.d` **replaces** the base image's
own init scripts (its `0001-install-pg_duckdb.sql` and the MotherDuck-only
`0002-enable-md-pg_duckdb.sql`). That's intended: our `10-parquet.sql` still
runs `CREATE EXTENSION`, and we don't use MotherDuck. If you prefer to keep
the image's baked scripts, drop the `:/docker-entrypoint-initdb.d` line — the
`Dockerfile` already bakes `10-parquet.sql` in.
- `:ro` keeps the DB from ever mutating the app's cache. In prod the app writes
the cache to the `appdata` volume; point this bind at wherever that lives on
the host if you want the DB to see the live cache rather than the repo copy.
- Init scripts only run when `PGDATA` is empty. On an **existing** database,
enable it once by hand:
```
docker compose exec db psql -U thermograph -d thermograph \
-c 'CREATE EXTENSION IF NOT EXISTS pg_duckdb;'
```
## Usage — real cache file at `/parquet/…`
Cache columns: `date, tmax, tmin, precip, wind, gust, humid, fmax, fmin, feels`.
pg_duckdb ≥ 0.3 uses the `r['colname']` subscript syntax (not
`AS t(col type, …)`).
```sql
-- one cell, first rows
SELECT r['date'] AS date, r['tmax'] AS tmax, r['tmin'] AS tmin, r['precip'] AS precip
FROM read_parquet('/parquet/1026_-2857.parquet') r
ORDER BY r['date'] LIMIT 5;
-- per-year analytics over one cell
SELECT EXTRACT(YEAR FROM r['date']::timestamp) AS yr,
ROUND(AVG(r['tmax'])::numeric, 1) AS avg_tmax,
MAX(r['tmax']) AS record_high
FROM read_parquet('/parquet/1026_-2857.parquet') r
WHERE r['date'] >= '2020-01-01'
GROUP BY yr ORDER BY yr;
-- glob across every cached cell (union_by_name handles the _rf / _forecast
-- files whose column sets differ)
SELECT COUNT(*) FROM read_parquet('/parquet/*.parquet', union_by_name := true) r;
```
## Proof of work (verified 2026-07-19)
Built `deploy/db/Dockerfile.db`, ran a throwaway container with the real
`data/cache` bind-mounted read-only at `/parquet`, then:
```
$ psql -c "SELECT version();"
PostgreSQL 18.1 (Debian 18.1-1.pgdg12+2) on x86_64-pc-linux-gnu ...
$ psql -c "SELECT extname, extversion FROM pg_extension WHERE extname='pg_duckdb';"
extname | extversion
-----------+------------
pg_duckdb | 1.1.0
$ psql -c "SELECT COUNT(*) FROM read_parquet('/parquet/1026_-2857.parquet') r;"
row_count
-----------
16982
$ psql -c "SELECT r['date'] AS date, r['tmax'] AS tmax, r['tmin'] AS tmin,
r['precip'] AS precip, r['humid'] AS humid
FROM read_parquet('/parquet/1026_-2857.parquet') r
ORDER BY r['date'] LIMIT 5;"
date | tmax | tmin | precip | humid
---------------------+------+------+--------+-------
1980-01-01 00:00:00 | 56.2 | 34.7 | 0 | 73
1980-01-02 00:00:00 | 63.8 | 38 | 0 | 79
1980-01-03 00:00:00 | 60.1 | 46.1 | 0.315 | 83
1980-01-04 00:00:00 | 51.7 | 40 | 0 | 68
1980-01-05 00:00:00 | 56.5 | 33.7 | 0 | 78
$ psql -c "SELECT EXTRACT(YEAR FROM r['date']::timestamp) AS yr,
ROUND(AVG(r['tmax'])::numeric,1) AS avg_tmax, MAX(r['tmax']) AS record_high
FROM read_parquet('/parquet/1026_-2857.parquet') r
WHERE r['date'] >= '2020-01-01' GROUP BY yr ORDER BY yr;"
yr | avg_tmax | record_high
------+----------+-------------
2020 | 78.8 | 99
2021 | 77.0 | 93
2022 | 79.2 | 101.4
2023 | 80.8 | 107.1
2024 | 79.8 | 97.5
2025 | 80.0 | 99.4
2026 | 77.8 | 95.6
$ psql -c "SELECT COUNT(*) FROM read_parquet('/parquet/*.parquet', union_by_name := true) r;"
count
--------
525421 -- rows across all 45 cached cells
```

15
deploy/db/init/10-parquet.sql Normal file
View file

@ -0,0 +1,15 @@
-- Enable pg_duckdb so the database can read the app's Parquet climate cache
-- directly (ad-hoc analytics via DuckDB's read_parquet / read_csv / etc.).
--
-- This runs once, on first cluster init (empty PGDATA), from
-- /docker-entrypoint-initdb.d. Because the compose `db` service keeps its data
-- on a persistent named volume, init scripts do NOT re-run on an existing
-- database -- to enable pg_duckdb on a DB that was created before this image,
-- run it by hand:
--
-- docker compose exec db psql -U thermograph -d thermograph \
-- -c 'CREATE EXTENSION IF NOT EXISTS pg_duckdb;'
--
-- pg_duckdb requires shared_preload_libraries='pg_duckdb'; the base image's
-- postgresql.conf.sample already sets it, so it is active on fresh init.
CREATE EXTENSION IF NOT EXISTS pg_duckdb;

71
deploy/deploy.sh
View file

@ -1,66 +1,58 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Pull the latest code and restart Thermograph. Run on the VPS — the GitHub # Pull the latest code and roll the docker-compose stack (app + PostgreSQL). Run
# Actions workflow invokes this over SSH, and you can run it by hand too. # on the VPS — the GitHub Actions workflow invokes this over SSH, and you can run
# it by hand too.
# #
# ssh deploy@vps '/opt/thermograph/deploy/deploy.sh' # ssh deploy@vps '/opt/thermograph/deploy/deploy.sh'
set -euo pipefail set -euo pipefail
APP_DIR="${APP_DIR:-/opt/thermograph}" APP_DIR="${APP_DIR:-/opt/thermograph}"
BRANCH="${BRANCH:-main}" BRANCH="${BRANCH:-main}"
HEALTH_PORT="${HEALTH_PORT:-8137}"
cd "$APP_DIR" cd "$APP_DIR"
# Secrets (POSTGRES_PASSWORD, VAPID keys, AUTH_SECRET, ...) drive compose
# interpolation and are also loaded into the app container via env_file. Source
# them here so a by-hand run interpolates the same as the systemd unit does.
set -a; . /etc/thermograph.env 2>/dev/null || true; set +a
# Pre-warm the ~750 city-page archives so /climate pages serve from cache and a # Pre-warm the ~750 city-page archives so /climate pages serve from cache and a
# search-engine crawl never bursts the archive API quota. Detached + backgrounded # search-engine crawl never bursts the archive API quota. Detached inside the app
# so it never blocks the deploy or health check; idempotent (skips already-cached # container (compose exec -d), idempotent (skips already-cached cells), so it
# cells), so it's cheap on every deploy after the first full warm. # never blocks the deploy or health check and is cheap on every deploy after the
# first full warm.
warm_city_archives() { warm_city_archives() {
mkdir -p "$APP_DIR/logs" echo "==> Warming city-page archives in the background (app:/app/logs/warm-cities.log)"
echo "==> Warming city-page archives in the background (logs/warm-cities.log)" docker compose exec -d app sh -c \
setsid nohup bash -c "cd '$APP_DIR/backend' && exec '$APP_DIR/.venv/bin/python' warm_cities.py --pace 2" \ 'python warm_cities.py --pace 2 >> /app/logs/warm-cities.log 2>&1' || true
</dev/null >"$APP_DIR/logs/warm-cities.log" 2>&1 &
} }
# Notify IndexNow (Bing / DuckDuckGo / Yandex) of the site's URLs, but only when # Notify IndexNow (Bing / DuckDuckGo / Yandex) of the site's URLs, but only when
# the set of pages actually changed (a new/removed city) — code-only deploys skip, # the set of pages actually changed (a new/removed city) — code-only deploys skip.
# so we don't re-blast ~14k URLs every push. Best-effort: never fails the deploy. # Best-effort: never fails the deploy.
# Sourcing the env matches the key the running service serves at /{key}.txt.
ping_indexnow() { ping_indexnow() {
echo "==> Pinging IndexNow (only if the URL set changed)" echo "==> Pinging IndexNow (only if the URL set changed)"
( set -a; . /etc/thermograph.env 2>/dev/null || true; set +a local base="${THERMOGRAPH_BASE_URL:-https://thermograph.org}"
base="${THERMOGRAPH_BASE_URL:-https://thermograph.org}" docker compose exec -T app python indexnow.py --if-changed "$base" \
cd "$APP_DIR/backend" && "$APP_DIR/.venv/bin/python" indexnow.py --if-changed "$base" || echo "!! IndexNow ping failed (non-fatal)" >&2
) || echo "!! IndexNow ping failed (non-fatal)" >&2
} }
echo "==> Fetching $BRANCH" echo "==> Fetching $BRANCH"
git fetch --prune origin "$BRANCH" git fetch --prune origin "$BRANCH"
git reset --hard "origin/$BRANCH" git reset --hard "origin/$BRANCH"
echo "==> Installing dependencies" echo "==> Building images"
if [ ! -d .venv ]; then docker compose build
python3 -m venv .venv
fi
.venv/bin/pip install --upgrade pip -q
.venv/bin/pip install -r backend/requirements.txt -q
# Apply any pending accounts-DB schema migrations before the new code starts, so # Schema migrations run inside the app container's entrypoint (alembic upgrade
# it never queries a column an older database lacks. Idempotent and tracked, so # head) before uvicorn starts, so there's no separate migrate step or service
# every deploy can run it; sourcing the env picks up a THERMOGRAPH_ACCOUNTS_DB # restart here — compose owns the process model.
# override if one is set. Runs as the deploy user, who owns the data dir. echo "==> Starting stack"
echo "==> Applying database migrations" docker compose up -d
( set -a; . /etc/thermograph.env 2>/dev/null || true; set +a
"$APP_DIR/.venv/bin/python" "$APP_DIR/deploy/migrate-db.py"
)
echo "==> Restarting service"
sudo systemctl restart thermograph
echo "==> Health check" echo "==> Health check"
PORT="$(sed -n 's/^PORT=//p' /etc/thermograph.env)"; PORT="${PORT:-8137}" url="http://127.0.0.1:${HEALTH_PORT}/"
BASE="$(sed -n 's/^THERMOGRAPH_BASE=//p' /etc/thermograph.env)"; BASE="${BASE:-/}" for i in $(seq 1 30); do
# Normalize: no double slash, allow root.
url="http://127.0.0.1:${PORT}${BASE%/}/"
for i in $(seq 1 15); do
if curl -fsS -o /dev/null "$url"; then if curl -fsS -o /dev/null "$url"; then
echo "==> OK: $url is serving" echo "==> OK: $url is serving"
warm_city_archives warm_city_archives
@ -70,5 +62,6 @@ for i in $(seq 1 15); do
sleep 1 sleep 1
done done
echo "!! Health check failed for $url" >&2 echo "!! Health check failed for $url" >&2
sudo systemctl status thermograph --no-pager -l || true docker compose ps || true
docker compose logs --tail=50 app || true
exit 1 exit 1

43
deploy/thermograph.env.example
View file

@ -1,17 +1,44 @@
# Copy to /etc/thermograph.env on the VPS and edit. # Copy to /etc/thermograph.env on the VPS and edit.
# Read by the systemd unit (EnvironmentFile). # Read by the systemd unit (EnvironmentFile) so `docker compose up` can interpolate
# it, AND loaded into the app container (env_file in docker-compose.yml). Anything
# secret the app needs — Postgres password, VAPID keys, auth secret — belongs here.
# Port uvicorn binds on loopback. Caddy proxies to this. Keep 8137 unless it clashes. # Port uvicorn binds inside the container. The compose stack publishes it on the
# host loopback (127.0.0.1:8137) for Caddy to proxy to. Keep 8137.
PORT=8137 PORT=8137
# --- PostgreSQL (docker-compose stack) ------------------------------------------
# The app and Postgres run as a docker-compose stack (see docker-compose.yml).
# POSTGRES_PASSWORD is the database password: compose uses it to initialize the
# postgres container AND to build the app's THERMOGRAPH_DATABASE_URL. It MUST be
# set here (the systemd unit sources this file so `docker compose up` can
# interpolate it). Change it from the default before the first `up`.
POSTGRES_PASSWORD=change-me
# The app's compose service already builds THERMOGRAPH_DATABASE_URL from
# POSTGRES_PASSWORD, so you normally DON'T need this. It's here for reference and
# for running the app outside compose against the same DB (keep the password in
# sync with POSTGRES_PASSWORD above).
#THERMOGRAPH_DATABASE_URL=postgresql+asyncpg://thermograph:change-me@db:5432/thermograph
# Mark the session cookie Secure — required behind Caddy's HTTPS. Set to 1 in prod;
# leave unset only for plain-HTTP LAN dev (a Secure cookie is never sent over HTTP).
THERMOGRAPH_COOKIE_SECURE=1
# Pin these too (see their own sections below), so container restarts don't rotate
# them: THERMOGRAPH_AUTH_SECRET (else every emailed confirm/reset link breaks on
# restart) and THERMOGRAPH_VAPID_PRIVATE_KEY / _PUBLIC_KEY (else every existing push
# subscription silently stops delivering). The data dir persists on the appdata
# volume, but pinning here is the safe default.
# Number of uvicorn worker processes. More than 1 stops a single slow upstream fetch # Number of uvicorn worker processes. More than 1 stops a single slow upstream fetch
# (e.g. a cache-miss weather lookup) from blocking every other request — the cause of # (e.g. a cache-miss weather lookup) from blocking every other request — the cause of
# past brief outages. Prod runs 3; leave unset (defaults to 1) on a small box. Workers # past brief outages. Prod runs 4 (the compose app service also defaults to WORKERS=4);
# share one metrics store (THERMOGRAPH_METRICS_DB, defaulted in the systemd unit) so the # leave unset (defaults to 1) on a small box. Workers elect one leader for the
# ops dashboard still sees the whole picture, and elect one leader for the subscription # subscription notifier via a lockfile (THERMOGRAPH_SINGLETON_LOCK, set by the compose
# notifier via a lockfile (THERMOGRAPH_SINGLETON_LOCK, also defaulted in the unit) so its # app service to /app/data/notifier.lock) so its timer-driven upstream sweep runs once,
# timer-driven upstream sweep runs once, not once per worker. ~200 MB RAM per worker. # not once per worker. ~200 MB RAM per worker.
WORKERS=3 WORKERS=4
# Base path the app is served under. # Base path the app is served under.
# / -> app at the domain root (Thermograph owns the whole domain — # / -> app at the domain root (Thermograph owns the whole domain —

49
deploy/thermograph.service
View file

@ -1,41 +1,22 @@
[Unit] [Unit]
Description=Thermograph (FastAPI/uvicorn) Description=Thermograph docker-compose stack (app + PostgreSQL)
After=network-online.target # Docker must be up, and the network online, before compose can pull/interpolate.
After=docker.service network-online.target
Wants=network-online.target Wants=network-online.target
Requires=docker.service
[Service] [Service]
Type=exec # A thin manager for the compose stack: `up -d` starts it and returns, the unit
User=deploy # stays "active" (RemainAfterExit) so `systemctl restart thermograph` re-runs it.
Group=deploy # Compose owns container ordering and restarts via depends_on + restart policies.
# The repo checkout. uvicorn is run from backend/ so `app:app` resolves. Type=oneshot
WorkingDirectory=/opt/thermograph/backend RemainAfterExit=yes
# Defaults; /etc/thermograph.env (below) overrides. WORKERS is how many uvicorn WorkingDirectory=/opt/thermograph
# worker processes to run — prod sets WORKERS=3 there; a single-worker box can leave # POSTGRES_PASSWORD (and the other secrets) are needed at `docker compose`
# it. Multiple workers each keep their own in-process state, so metrics go to a shared # interpolation time. The leading `-` makes a missing file non-fatal.
# SQLite DB (THERMOGRAPH_METRICS_DB) that every worker tallies into — otherwise the ops EnvironmentFile=-/etc/thermograph.env
# dashboard, polling one random worker, would see only a fraction of the traffic. ExecStart=/usr/bin/docker compose up -d
Environment=WORKERS=1 ExecStop=/usr/bin/docker compose down
Environment=THERMOGRAPH_METRICS_DB=/opt/thermograph/data/metrics.db
# Elect one worker to run the subscription notifier (its timer-driven upstream sweep
# must run once across the deploy, not once per worker — else it multiplies the
# Open-Meteo quota use). Workers race for this lockfile; the winner runs the notifier.
Environment=THERMOGRAPH_SINGLETON_LOCK=/opt/thermograph/data/notifier.lock
# THERMOGRAPH_BASE, PORT, WORKERS, etc. See deploy/thermograph.env.example.
EnvironmentFile=/etc/thermograph.env
# Clear the shared metrics DB on each (re)start so the dashboard's "since start" tallies
# reflect the running workers, matching how the old in-process counters behaved.
ExecStartPre=/usr/bin/rm -f /opt/thermograph/data/metrics.db /opt/thermograph/data/metrics.db-wal /opt/thermograph/data/metrics.db-shm
# Bind loopback only — Caddy terminates TLS and reverse-proxies to us.
ExecStart=/opt/thermograph/.venv/bin/uvicorn app:app --host 127.0.0.1 --port ${PORT} --workers ${WORKERS}
Restart=on-failure
RestartSec=2
# Hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ProtectHome=read-only
# The parquet cache must stay writable across deploys.
ReadWritePaths=/opt/thermograph/data /opt/thermograph/logs
[Install] [Install]
WantedBy=multi-user.target WantedBy=multi-user.target

86
docker-compose.yml Normal file
View file

@ -0,0 +1,86 @@
# Thermograph production stack: the FastAPI app plus its PostgreSQL 18 database.
#
# docker compose up -d --build # or: make up
#
# POSTGRES_PASSWORD must be set at `docker compose` time — compose reads it from
# the repo-root .env for local runs (copy .env.example -> .env), and in prod the
# systemd unit's EnvironmentFile=/etc/thermograph.env puts it in the environment
# so `docker compose up` can interpolate it. It is used BOTH to initialize the db
# container and to build the app's THERMOGRAPH_DATABASE_URL below.
services:
db:
# PostgreSQL 18 + pg_duckdb (deploy/db/Dockerfile.db, FROM pgduckdb/pgduckdb:18-…,
# which IS PostgreSQL 18). Lets the DB read the app's parquet climate cache
# directly, e.g. SELECT * FROM read_parquet('/parquet/cache/*.parquet'). The init
# script CREATE EXTENSIONs pg_duckdb on a fresh volume.
build:
context: .
dockerfile: deploy/db/Dockerfile.db
environment:
POSTGRES_USER: thermograph
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}
POSTGRES_DB: thermograph
volumes:
# Mount the volume at the PARENT of the data dir and let the PG18 image use
# its own versioned subdir (/var/lib/postgresql/18/docker). Pinning PGDATA
# directly at the mountpoint trips an initdb chmod on some Docker setups; the
# whole tree still persists on the named volume this way.
- pgdata:/var/lib/postgresql
# Read the app's live parquet cache (the same appdata volume the app writes,
# mounted read-only) — query it under /parquet/cache/*.parquet.
- appdata:/parquet:ro
- ./deploy/db/init:/docker-entrypoint-initdb.d
healthcheck:
test: ["CMD-SHELL", "pg_isready -U thermograph -d thermograph"]
interval: 5s
timeout: 5s
retries: 10
# Cap the DB at 2 CPUs. Compose v2 honors the top-level `cpus:`; the
# deploy.resources block is the Swarm-style equivalent, kept for parity.
cpus: 2.0
deploy:
resources:
limits:
cpus: "2.0"
restart: unless-stopped
# No host port on purpose: the app reaches Postgres as db:5432 on the
# compose network. Nothing outside the stack should touch the database.
app:
build: .
depends_on:
db:
condition: service_healthy
environment:
# Built from POSTGRES_PASSWORD; this `environment` value wins over anything
# in env_file, so the URL always matches the db container's password.
THERMOGRAPH_DATABASE_URL: postgresql+asyncpg://thermograph:${POSTGRES_PASSWORD}@db:5432/thermograph
THERMOGRAPH_BASE: /
PORT: 8137
WORKERS: 4
# One worker wins this lock and runs the subscription notifier / homepage
# sweep; it lives on the appdata volume so it's shared across workers.
THERMOGRAPH_SINGLETON_LOCK: /app/data/notifier.lock
# Prod secrets live in /etc/thermograph.env: POSTGRES_PASSWORD,
# THERMOGRAPH_AUTH_SECRET, THERMOGRAPH_VAPID_PRIVATE_KEY/_PUBLIC_KEY,
# THERMOGRAPH_COOKIE_SECURE=1, mail/Discord keys, ... (see
# deploy/thermograph.env.example). `required: false` so local `docker compose
# up` works without that file — it reads POSTGRES_PASSWORD from repo-root .env.
env_file:
- path: /etc/thermograph.env
required: false
volumes:
# Parquet cache, notifier.lock, homepage.json, vapid.json persist here.
- appdata:/app/data
- applogs:/app/logs
cpus: 4.0
ports:
# Loopback only — host Caddy terminates TLS and reverse-proxies to this.
- "127.0.0.1:8137:8137"
restart: unless-stopped
volumes:
pgdata: {}
appdata: {}
applogs: {}