Containerize the app and move the databases to PostgreSQL 18 (#220)
Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the
data layer on Postgres, while keeping the test suite on SQLite.
- accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write +
read-only asyncpg pair (the RO engine pins read-only transactions, used by the
pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the
SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL
is unset). models.py: boolean server_default -> sa.false().
- store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg
when configured, else the existing raw-sqlite3 paths byte-for-byte; sync
interfaces and every fail-soft contract preserved.
- Alembic (backend/alembic/) manages the accounts schema; the container entrypoint
runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py
copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced),
skips access_token, and resets identity sequences.
- Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137)
and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet
climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet').
- deploy.sh/thermograph.service rewired to manage the compose stack; env example,
Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook.
Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was
verified via docker compose: alembic migrations, register/login, the RO endpoint,
store/metrics round-trips, and the accounts data migration.
2026-07-20 06:28:23 +00:00
|
|
|
#!/usr/bin/env bash
|
Add entrypoint one-shot migrate mode and a Swarm secrets shim (#231)
Two changes to deploy/entrypoint.sh, both needed before the app can run
under Docker Swarm without re-triggering known incidents:
One-shot migrate mode: `entrypoint.sh migrate` (or THERMOGRAPH_MIGRATE_ONLY=1)
runs the Alembic migration and exits, for a Swarm one-shot task that brings
the schema to head once rather than racing it across every web replica's
boot. RUN_MIGRATIONS=0 skips the inline migrate for a deploy that runs the
one-shot task separately. Default (unset) keeps today's behavior unchanged:
every boot migrates itself before serving.
Secrets shim: the app reads config from os.environ everywhere (an unset
THERMOGRAPH_AUTH_SECRET falls back to a random per-process value; an unset
VAPID key pair is freshly minted) - Swarm `secrets:` mount each secret as a
FILE under /run/secrets/ instead, which the app would never see. Before
alembic/uvicorn, mechanically export each /run/secrets/<name> whose
uppercased name is a THERMOGRAPH_* var, unless already set in the process
env; THERMOGRAPH_DATABASE_URL is built from a postgres_password secret the
same way compose's own interpolation builds it today.
Verified against fake alembic/uvicorn binaries in a throwaway container:
default boot, migrate-only mode (both trigger forms), RUN_MIGRATIONS=0,
secrets populating unset vars, an existing env var beating its secret file,
and no /run/secrets present at all (today's plain compose path, unaffected).
shellcheck clean.
2026-07-21 00:18:05 +00:00
|
|
|
# Container entrypoint: load secrets, bring the accounts schema up to head, then serve.
|
Containerize the app and move the databases to PostgreSQL 18 (#220)
Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the
data layer on Postgres, while keeping the test suite on SQLite.
- accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write +
read-only asyncpg pair (the RO engine pins read-only transactions, used by the
pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the
SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL
is unset). models.py: boolean server_default -> sa.false().
- store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg
when configured, else the existing raw-sqlite3 paths byte-for-byte; sync
interfaces and every fail-soft contract preserved.
- Alembic (backend/alembic/) manages the accounts schema; the container entrypoint
runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py
copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced),
skips access_token, and resets identity sequences.
- Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137)
and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet
climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet').
- deploy.sh/thermograph.service rewired to manage the compose stack; env example,
Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook.
Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was
verified via docker compose: alembic migrations, register/login, the RO endpoint,
store/metrics round-trips, and the accounts data migration.
2026-07-20 06:28:23 +00:00
|
|
|
#
|
|
|
|
|
# Alembic runs before uvicorn so a fresh Postgres volume gets its tables. It is
|
|
|
|
|
# retried because `db` can still be finishing startup even though compose waits on
|
|
|
|
|
# its healthcheck — a cold volume's first connection may race the server becoming
|
|
|
|
|
# ready to accept the migration.
|
|
|
|
|
set -euo pipefail
|
|
|
|
|
|
2026-07-21 22:57:17 +00:00
|
|
|
cd /app
|
Containerize the app and move the databases to PostgreSQL 18 (#220)
Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the
data layer on Postgres, while keeping the test suite on SQLite.
- accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write +
read-only asyncpg pair (the RO engine pins read-only transactions, used by the
pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the
SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL
is unset). models.py: boolean server_default -> sa.false().
- store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg
when configured, else the existing raw-sqlite3 paths byte-for-byte; sync
interfaces and every fail-soft contract preserved.
- Alembic (backend/alembic/) manages the accounts schema; the container entrypoint
runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py
copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced),
skips access_token, and resets identity sequences.
- Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137)
and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet
climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet').
- deploy.sh/thermograph.service rewired to manage the compose stack; env example,
Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook.
Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was
verified via docker compose: alembic migrations, register/login, the RO endpoint,
store/metrics round-trips, and the accounts data migration.
2026-07-20 06:28:23 +00:00
|
|
|
|
Add entrypoint one-shot migrate mode and a Swarm secrets shim (#231)
Two changes to deploy/entrypoint.sh, both needed before the app can run
under Docker Swarm without re-triggering known incidents:
One-shot migrate mode: `entrypoint.sh migrate` (or THERMOGRAPH_MIGRATE_ONLY=1)
runs the Alembic migration and exits, for a Swarm one-shot task that brings
the schema to head once rather than racing it across every web replica's
boot. RUN_MIGRATIONS=0 skips the inline migrate for a deploy that runs the
one-shot task separately. Default (unset) keeps today's behavior unchanged:
every boot migrates itself before serving.
Secrets shim: the app reads config from os.environ everywhere (an unset
THERMOGRAPH_AUTH_SECRET falls back to a random per-process value; an unset
VAPID key pair is freshly minted) - Swarm `secrets:` mount each secret as a
FILE under /run/secrets/ instead, which the app would never see. Before
alembic/uvicorn, mechanically export each /run/secrets/<name> whose
uppercased name is a THERMOGRAPH_* var, unless already set in the process
env; THERMOGRAPH_DATABASE_URL is built from a postgres_password secret the
same way compose's own interpolation builds it today.
Verified against fake alembic/uvicorn binaries in a throwaway container:
default boot, migrate-only mode (both trigger forms), RUN_MIGRATIONS=0,
secrets populating unset vars, an existing env var beating its secret file,
and no /run/secrets present at all (today's plain compose path, unaffected).
shellcheck clean.
2026-07-21 00:18:05 +00:00
|
|
|
# --- Swarm secrets shim ------------------------------------------------------
|
|
|
|
|
# The app reads config from os.environ everywhere (users.py falls back to a random
|
|
|
|
|
# per-process AUTH_SECRET if THERMOGRAPH_AUTH_SECRET is unset; push.py mints a new
|
|
|
|
|
# VAPID keypair if the VAPID env vars are unset) — compose delivers config that way
|
|
|
|
|
# today via `environment:`/`env_file`. Swarm `secrets:` instead mount each secret as
|
|
|
|
|
# a FILE under /run/secrets/<name>, which the app would never see: without this shim,
|
|
|
|
|
# moving to Swarm secrets silently re-triggers those exact incidents (every session
|
|
|
|
|
# invalidated, every push subscription broken) on the next restart.
|
|
|
|
|
#
|
|
|
|
|
# Mechanical mapping, so a new THERMOGRAPH_* var never needs an entrypoint edit: a
|
|
|
|
|
# secret file's UPPERCASED name IS the env var it becomes — e.g. a secret named
|
|
|
|
|
# thermograph_auth_secret -> THERMOGRAPH_AUTH_SECRET — skipping anything that
|
|
|
|
|
# doesn't uppercase to a THERMOGRAPH_* name (postgres_password is handled below
|
|
|
|
|
# instead). Never overrides an already-set var (environment:/env_file always wins).
|
|
|
|
|
# Secret names must be valid shell-identifier characters (letters, digits,
|
|
|
|
|
# underscore) to become a var name.
|
|
|
|
|
if [ -d /run/secrets ]; then
|
|
|
|
|
for f in /run/secrets/*; do
|
|
|
|
|
[ -f "$f" ] || continue
|
|
|
|
|
var=$(basename "$f" | tr '[:lower:]' '[:upper:]')
|
|
|
|
|
case "$var" in
|
|
|
|
|
THERMOGRAPH_*)
|
|
|
|
|
if [ -z "${!var:-}" ]; then
|
|
|
|
|
export "$var=$(cat "$f")"
|
|
|
|
|
fi
|
|
|
|
|
;;
|
|
|
|
|
esac
|
|
|
|
|
done
|
|
|
|
|
|
|
|
|
|
# THERMOGRAPH_DATABASE_URL is normally built by compose's own interpolation
|
|
|
|
|
# (POSTGRES_PASSWORD baked into docker-compose.yml's `environment:` block) — Swarm
|
|
|
|
|
# secrets have no interpolation, so build the same URL here from the
|
|
|
|
|
# `postgres_password` secret when the URL isn't already set some other way.
|
|
|
|
|
if [ -z "${THERMOGRAPH_DATABASE_URL:-}" ] && [ -f /run/secrets/postgres_password ]; then
|
|
|
|
|
pg_pw=$(cat /run/secrets/postgres_password)
|
|
|
|
|
export THERMOGRAPH_DATABASE_URL="postgresql+asyncpg://thermograph:${pg_pw}@db:5432/thermograph"
|
Containerize the app and move the databases to PostgreSQL 18 (#220)
Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the
data layer on Postgres, while keeping the test suite on SQLite.
- accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write +
read-only asyncpg pair (the RO engine pins read-only transactions, used by the
pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the
SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL
is unset). models.py: boolean server_default -> sa.false().
- store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg
when configured, else the existing raw-sqlite3 paths byte-for-byte; sync
interfaces and every fail-soft contract preserved.
- Alembic (backend/alembic/) manages the accounts schema; the container entrypoint
runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py
copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced),
skips access_token, and resets identity sequences.
- Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137)
and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet
climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet').
- deploy.sh/thermograph.service rewired to manage the compose stack; env example,
Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook.
Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was
verified via docker compose: alembic migrations, register/login, the RO endpoint,
store/metrics round-trips, and the accounts data migration.
2026-07-20 06:28:23 +00:00
|
|
|
fi
|
Add entrypoint one-shot migrate mode and a Swarm secrets shim (#231)
Two changes to deploy/entrypoint.sh, both needed before the app can run
under Docker Swarm without re-triggering known incidents:
One-shot migrate mode: `entrypoint.sh migrate` (or THERMOGRAPH_MIGRATE_ONLY=1)
runs the Alembic migration and exits, for a Swarm one-shot task that brings
the schema to head once rather than racing it across every web replica's
boot. RUN_MIGRATIONS=0 skips the inline migrate for a deploy that runs the
one-shot task separately. Default (unset) keeps today's behavior unchanged:
every boot migrates itself before serving.
Secrets shim: the app reads config from os.environ everywhere (an unset
THERMOGRAPH_AUTH_SECRET falls back to a random per-process value; an unset
VAPID key pair is freshly minted) - Swarm `secrets:` mount each secret as a
FILE under /run/secrets/ instead, which the app would never see. Before
alembic/uvicorn, mechanically export each /run/secrets/<name> whose
uppercased name is a THERMOGRAPH_* var, unless already set in the process
env; THERMOGRAPH_DATABASE_URL is built from a postgres_password secret the
same way compose's own interpolation builds it today.
Verified against fake alembic/uvicorn binaries in a throwaway container:
default boot, migrate-only mode (both trigger forms), RUN_MIGRATIONS=0,
secrets populating unset vars, an existing env var beating its secret file,
and no /run/secrets present at all (today's plain compose path, unaffected).
shellcheck clean.
2026-07-21 00:18:05 +00:00
|
|
|
fi
|
|
|
|
|
|
|
|
|
|
run_migrations() {
|
|
|
|
|
local max=10 tries=0
|
|
|
|
|
echo "==> Running database migrations (alembic upgrade head)"
|
|
|
|
|
until alembic upgrade head; do
|
|
|
|
|
tries=$((tries + 1))
|
|
|
|
|
if [ "$tries" -ge "$max" ]; then
|
|
|
|
|
echo "!! alembic upgrade head failed after ${max} attempts; giving up" >&2
|
|
|
|
|
exit 1
|
|
|
|
|
fi
|
|
|
|
|
echo " migration attempt ${tries}/${max} failed; retrying in 3s..." >&2
|
|
|
|
|
sleep 3
|
|
|
|
|
done
|
|
|
|
|
}
|
|
|
|
|
|
2026-07-23 21:20:35 +00:00
|
|
|
# The lake role serves the ERA5 bucket only — no database, so no migrations: a
|
|
|
|
|
# lake replica must come up (and stay up) even when Postgres is down.
|
|
|
|
|
if [ "${THERMOGRAPH_ROLE:-}" = "lake" ]; then
|
|
|
|
|
echo "==> Starting lake service on 0.0.0.0:${PORT:-8141} with ${WORKERS:-1} worker(s)"
|
|
|
|
|
exec uvicorn lake_app:app --host 0.0.0.0 --port "${PORT:-8141}" --workers "${WORKERS:-1}"
|
|
|
|
|
fi
|
|
|
|
|
|
Add entrypoint one-shot migrate mode and a Swarm secrets shim (#231)
Two changes to deploy/entrypoint.sh, both needed before the app can run
under Docker Swarm without re-triggering known incidents:
One-shot migrate mode: `entrypoint.sh migrate` (or THERMOGRAPH_MIGRATE_ONLY=1)
runs the Alembic migration and exits, for a Swarm one-shot task that brings
the schema to head once rather than racing it across every web replica's
boot. RUN_MIGRATIONS=0 skips the inline migrate for a deploy that runs the
one-shot task separately. Default (unset) keeps today's behavior unchanged:
every boot migrates itself before serving.
Secrets shim: the app reads config from os.environ everywhere (an unset
THERMOGRAPH_AUTH_SECRET falls back to a random per-process value; an unset
VAPID key pair is freshly minted) - Swarm `secrets:` mount each secret as a
FILE under /run/secrets/ instead, which the app would never see. Before
alembic/uvicorn, mechanically export each /run/secrets/<name> whose
uppercased name is a THERMOGRAPH_* var, unless already set in the process
env; THERMOGRAPH_DATABASE_URL is built from a postgres_password secret the
same way compose's own interpolation builds it today.
Verified against fake alembic/uvicorn binaries in a throwaway container:
default boot, migrate-only mode (both trigger forms), RUN_MIGRATIONS=0,
secrets populating unset vars, an existing env var beating its secret file,
and no /run/secrets present at all (today's plain compose path, unaffected).
shellcheck clean.
2026-07-21 00:18:05 +00:00
|
|
|
# One-shot migrate mode (`entrypoint.sh migrate`, or THERMOGRAPH_MIGRATE_ONLY=1): run
|
2026-07-21 22:57:17 +00:00
|
|
|
# the migration and exit — a one-shot task, so it runs once against the DB rather
|
|
|
|
|
# than racing per web replica.
|
Add entrypoint one-shot migrate mode and a Swarm secrets shim (#231)
Two changes to deploy/entrypoint.sh, both needed before the app can run
under Docker Swarm without re-triggering known incidents:
One-shot migrate mode: `entrypoint.sh migrate` (or THERMOGRAPH_MIGRATE_ONLY=1)
runs the Alembic migration and exits, for a Swarm one-shot task that brings
the schema to head once rather than racing it across every web replica's
boot. RUN_MIGRATIONS=0 skips the inline migrate for a deploy that runs the
one-shot task separately. Default (unset) keeps today's behavior unchanged:
every boot migrates itself before serving.
Secrets shim: the app reads config from os.environ everywhere (an unset
THERMOGRAPH_AUTH_SECRET falls back to a random per-process value; an unset
VAPID key pair is freshly minted) - Swarm `secrets:` mount each secret as a
FILE under /run/secrets/ instead, which the app would never see. Before
alembic/uvicorn, mechanically export each /run/secrets/<name> whose
uppercased name is a THERMOGRAPH_* var, unless already set in the process
env; THERMOGRAPH_DATABASE_URL is built from a postgres_password secret the
same way compose's own interpolation builds it today.
Verified against fake alembic/uvicorn binaries in a throwaway container:
default boot, migrate-only mode (both trigger forms), RUN_MIGRATIONS=0,
secrets populating unset vars, an existing env var beating its secret file,
and no /run/secrets present at all (today's plain compose path, unaffected).
shellcheck clean.
2026-07-21 00:18:05 +00:00
|
|
|
if [ "${1:-}" = "migrate" ] || [ "${THERMOGRAPH_MIGRATE_ONLY:-0}" = "1" ]; then
|
|
|
|
|
run_migrations
|
|
|
|
|
exit 0
|
|
|
|
|
fi
|
|
|
|
|
|
|
|
|
|
# RUN_MIGRATIONS=0 skips the inline migrate below — for a deploy that runs the
|
|
|
|
|
# one-shot task above separately and doesn't want every web replica to also race it.
|
|
|
|
|
# Default (unset) keeps today's behavior: every boot brings its own schema to head.
|
|
|
|
|
if [ "${RUN_MIGRATIONS:-1}" != "0" ]; then
|
|
|
|
|
run_migrations
|
|
|
|
|
fi
|
Containerize the app and move the databases to PostgreSQL 18 (#220)
Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the
data layer on Postgres, while keeping the test suite on SQLite.
- accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write +
read-only asyncpg pair (the RO engine pins read-only transactions, used by the
pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the
SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL
is unset). models.py: boolean server_default -> sa.false().
- store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg
when configured, else the existing raw-sqlite3 paths byte-for-byte; sync
interfaces and every fail-soft contract preserved.
- Alembic (backend/alembic/) manages the accounts schema; the container entrypoint
runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py
copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced),
skips access_token, and resets identity sequences.
- Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137)
and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet
climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet').
- deploy.sh/thermograph.service rewired to manage the compose stack; env example,
Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook.
Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was
verified via docker compose: alembic migrations, register/login, the RO endpoint,
store/metrics round-trips, and the accounts data migration.
2026-07-20 06:28:23 +00:00
|
|
|
|
2026-07-21 22:57:17 +00:00
|
|
|
echo "==> Starting uvicorn on 0.0.0.0:${PORT:-8137} with ${WORKERS:-4} worker(s)"
|
2026-07-24 19:28:47 +00:00
|
|
|
# --no-access-log: the app's own request-logging middleware (audit.log_access,
|
|
|
|
|
# web/app.py) already writes a structured line per request; uvicorn's own access
|
|
|
|
|
# log just duplicated every one of them for no benefit.
|
|
|
|
|
exec uvicorn app:app --host 0.0.0.0 --port "${PORT:-8137}" --workers "${WORKERS:-4}" --no-access-log
|