thermograph/backend/deploy/entrypoint.sh

92 lines
4.2 KiB
Bash
Raw Normal View History

Containerize the app and move the databases to PostgreSQL 18 (#220) Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the data layer on Postgres, while keeping the test suite on SQLite. - accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write + read-only asyncpg pair (the RO engine pins read-only transactions, used by the pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL is unset). models.py: boolean server_default -> sa.false(). - store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg when configured, else the existing raw-sqlite3 paths byte-for-byte; sync interfaces and every fail-soft contract preserved. - Alembic (backend/alembic/) manages the accounts schema; the container entrypoint runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced), skips access_token, and resets identity sequences. - Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137) and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet'). - deploy.sh/thermograph.service rewired to manage the compose stack; env example, Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook. Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was verified via docker compose: alembic migrations, register/login, the RO endpoint, store/metrics round-trips, and the accounts data migration.
2026-07-20 06:28:23 +00:00
#!/usr/bin/env bash
Add entrypoint one-shot migrate mode and a Swarm secrets shim (#231) Two changes to deploy/entrypoint.sh, both needed before the app can run under Docker Swarm without re-triggering known incidents: One-shot migrate mode: `entrypoint.sh migrate` (or THERMOGRAPH_MIGRATE_ONLY=1) runs the Alembic migration and exits, for a Swarm one-shot task that brings the schema to head once rather than racing it across every web replica's boot. RUN_MIGRATIONS=0 skips the inline migrate for a deploy that runs the one-shot task separately. Default (unset) keeps today's behavior unchanged: every boot migrates itself before serving. Secrets shim: the app reads config from os.environ everywhere (an unset THERMOGRAPH_AUTH_SECRET falls back to a random per-process value; an unset VAPID key pair is freshly minted) - Swarm `secrets:` mount each secret as a FILE under /run/secrets/ instead, which the app would never see. Before alembic/uvicorn, mechanically export each /run/secrets/<name> whose uppercased name is a THERMOGRAPH_* var, unless already set in the process env; THERMOGRAPH_DATABASE_URL is built from a postgres_password secret the same way compose's own interpolation builds it today. Verified against fake alembic/uvicorn binaries in a throwaway container: default boot, migrate-only mode (both trigger forms), RUN_MIGRATIONS=0, secrets populating unset vars, an existing env var beating its secret file, and no /run/secrets present at all (today's plain compose path, unaffected). shellcheck clean.
2026-07-21 00:18:05 +00:00
# Container entrypoint: load secrets, bring the accounts schema up to head, then serve.
Containerize the app and move the databases to PostgreSQL 18 (#220) Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the data layer on Postgres, while keeping the test suite on SQLite. - accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write + read-only asyncpg pair (the RO engine pins read-only transactions, used by the pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL is unset). models.py: boolean server_default -> sa.false(). - store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg when configured, else the existing raw-sqlite3 paths byte-for-byte; sync interfaces and every fail-soft contract preserved. - Alembic (backend/alembic/) manages the accounts schema; the container entrypoint runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced), skips access_token, and resets identity sequences. - Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137) and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet'). - deploy.sh/thermograph.service rewired to manage the compose stack; env example, Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook. Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was verified via docker compose: alembic migrations, register/login, the RO endpoint, store/metrics round-trips, and the accounts data migration.
2026-07-20 06:28:23 +00:00
#
# Alembic runs before uvicorn so a fresh Postgres volume gets its tables. It is
# retried because `db` can still be finishing startup even though compose waits on
# its healthcheck — a cold volume's first connection may race the server becoming
# ready to accept the migration.
set -euo pipefail
cd /app
Containerize the app and move the databases to PostgreSQL 18 (#220) Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the data layer on Postgres, while keeping the test suite on SQLite. - accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write + read-only asyncpg pair (the RO engine pins read-only transactions, used by the pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL is unset). models.py: boolean server_default -> sa.false(). - store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg when configured, else the existing raw-sqlite3 paths byte-for-byte; sync interfaces and every fail-soft contract preserved. - Alembic (backend/alembic/) manages the accounts schema; the container entrypoint runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced), skips access_token, and resets identity sequences. - Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137) and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet'). - deploy.sh/thermograph.service rewired to manage the compose stack; env example, Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook. Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was verified via docker compose: alembic migrations, register/login, the RO endpoint, store/metrics round-trips, and the accounts data migration.
2026-07-20 06:28:23 +00:00
Add entrypoint one-shot migrate mode and a Swarm secrets shim (#231) Two changes to deploy/entrypoint.sh, both needed before the app can run under Docker Swarm without re-triggering known incidents: One-shot migrate mode: `entrypoint.sh migrate` (or THERMOGRAPH_MIGRATE_ONLY=1) runs the Alembic migration and exits, for a Swarm one-shot task that brings the schema to head once rather than racing it across every web replica's boot. RUN_MIGRATIONS=0 skips the inline migrate for a deploy that runs the one-shot task separately. Default (unset) keeps today's behavior unchanged: every boot migrates itself before serving. Secrets shim: the app reads config from os.environ everywhere (an unset THERMOGRAPH_AUTH_SECRET falls back to a random per-process value; an unset VAPID key pair is freshly minted) - Swarm `secrets:` mount each secret as a FILE under /run/secrets/ instead, which the app would never see. Before alembic/uvicorn, mechanically export each /run/secrets/<name> whose uppercased name is a THERMOGRAPH_* var, unless already set in the process env; THERMOGRAPH_DATABASE_URL is built from a postgres_password secret the same way compose's own interpolation builds it today. Verified against fake alembic/uvicorn binaries in a throwaway container: default boot, migrate-only mode (both trigger forms), RUN_MIGRATIONS=0, secrets populating unset vars, an existing env var beating its secret file, and no /run/secrets present at all (today's plain compose path, unaffected). shellcheck clean.
2026-07-21 00:18:05 +00:00
# --- Swarm secrets shim ------------------------------------------------------
# The app reads config from os.environ everywhere (users.py falls back to a random
# per-process AUTH_SECRET if THERMOGRAPH_AUTH_SECRET is unset; push.py mints a new
# VAPID keypair if the VAPID env vars are unset) — compose delivers config that way
# today via `environment:`/`env_file`. Swarm `secrets:` instead mount each secret as
# a FILE under /run/secrets/<name>, which the app would never see: without this shim,
# moving to Swarm secrets silently re-triggers those exact incidents (every session
# invalidated, every push subscription broken) on the next restart.
#
# Mechanical mapping, so a new THERMOGRAPH_* var never needs an entrypoint edit: a
# secret file's UPPERCASED name IS the env var it becomes — e.g. a secret named
# thermograph_auth_secret -> THERMOGRAPH_AUTH_SECRET — skipping anything that
# doesn't uppercase to a THERMOGRAPH_* name (postgres_password is handled below
# instead). Never overrides an already-set var (environment:/env_file always wins).
# Secret names must be valid shell-identifier characters (letters, digits,
# underscore) to become a var name.
if [ -d /run/secrets ]; then
for f in /run/secrets/*; do
[ -f "$f" ] || continue
var=$(basename "$f" | tr '[:lower:]' '[:upper:]')
case "$var" in
THERMOGRAPH_*)
if [ -z "${!var:-}" ]; then
export "$var=$(cat "$f")"
fi
;;
esac
done
# THERMOGRAPH_DATABASE_URL is normally built by compose's own interpolation
# (POSTGRES_PASSWORD baked into docker-compose.yml's `environment:` block) — Swarm
# secrets have no interpolation, so build the same URL here from the
# `postgres_password` secret when the URL isn't already set some other way.
if [ -z "${THERMOGRAPH_DATABASE_URL:-}" ] && [ -f /run/secrets/postgres_password ]; then
pg_pw=$(cat /run/secrets/postgres_password)
export THERMOGRAPH_DATABASE_URL="postgresql+asyncpg://thermograph:${pg_pw}@db:5432/thermograph"
Containerize the app and move the databases to PostgreSQL 18 (#220) Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the data layer on Postgres, while keeping the test suite on SQLite. - accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write + read-only asyncpg pair (the RO engine pins read-only transactions, used by the pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL is unset). models.py: boolean server_default -> sa.false(). - store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg when configured, else the existing raw-sqlite3 paths byte-for-byte; sync interfaces and every fail-soft contract preserved. - Alembic (backend/alembic/) manages the accounts schema; the container entrypoint runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced), skips access_token, and resets identity sequences. - Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137) and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet'). - deploy.sh/thermograph.service rewired to manage the compose stack; env example, Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook. Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was verified via docker compose: alembic migrations, register/login, the RO endpoint, store/metrics round-trips, and the accounts data migration.
2026-07-20 06:28:23 +00:00
fi
Add entrypoint one-shot migrate mode and a Swarm secrets shim (#231) Two changes to deploy/entrypoint.sh, both needed before the app can run under Docker Swarm without re-triggering known incidents: One-shot migrate mode: `entrypoint.sh migrate` (or THERMOGRAPH_MIGRATE_ONLY=1) runs the Alembic migration and exits, for a Swarm one-shot task that brings the schema to head once rather than racing it across every web replica's boot. RUN_MIGRATIONS=0 skips the inline migrate for a deploy that runs the one-shot task separately. Default (unset) keeps today's behavior unchanged: every boot migrates itself before serving. Secrets shim: the app reads config from os.environ everywhere (an unset THERMOGRAPH_AUTH_SECRET falls back to a random per-process value; an unset VAPID key pair is freshly minted) - Swarm `secrets:` mount each secret as a FILE under /run/secrets/ instead, which the app would never see. Before alembic/uvicorn, mechanically export each /run/secrets/<name> whose uppercased name is a THERMOGRAPH_* var, unless already set in the process env; THERMOGRAPH_DATABASE_URL is built from a postgres_password secret the same way compose's own interpolation builds it today. Verified against fake alembic/uvicorn binaries in a throwaway container: default boot, migrate-only mode (both trigger forms), RUN_MIGRATIONS=0, secrets populating unset vars, an existing env var beating its secret file, and no /run/secrets present at all (today's plain compose path, unaffected). shellcheck clean.
2026-07-21 00:18:05 +00:00
fi
run_migrations() {
local max=10 tries=0
echo "==> Running database migrations (alembic upgrade head)"
until alembic upgrade head; do
tries=$((tries + 1))
if [ "$tries" -ge "$max" ]; then
echo "!! alembic upgrade head failed after ${max} attempts; giving up" >&2
exit 1
fi
echo " migration attempt ${tries}/${max} failed; retrying in 3s..." >&2
sleep 3
done
}
# The lake role serves the ERA5 bucket only — no database, so no migrations: a
# lake replica must come up (and stay up) even when Postgres is down.
if [ "${THERMOGRAPH_ROLE:-}" = "lake" ]; then
echo "==> Starting lake service on 0.0.0.0:${PORT:-8141} with ${WORKERS:-1} worker(s)"
exec uvicorn lake_app:app --host 0.0.0.0 --port "${PORT:-8141}" --workers "${WORKERS:-1}"
fi
Add entrypoint one-shot migrate mode and a Swarm secrets shim (#231) Two changes to deploy/entrypoint.sh, both needed before the app can run under Docker Swarm without re-triggering known incidents: One-shot migrate mode: `entrypoint.sh migrate` (or THERMOGRAPH_MIGRATE_ONLY=1) runs the Alembic migration and exits, for a Swarm one-shot task that brings the schema to head once rather than racing it across every web replica's boot. RUN_MIGRATIONS=0 skips the inline migrate for a deploy that runs the one-shot task separately. Default (unset) keeps today's behavior unchanged: every boot migrates itself before serving. Secrets shim: the app reads config from os.environ everywhere (an unset THERMOGRAPH_AUTH_SECRET falls back to a random per-process value; an unset VAPID key pair is freshly minted) - Swarm `secrets:` mount each secret as a FILE under /run/secrets/ instead, which the app would never see. Before alembic/uvicorn, mechanically export each /run/secrets/<name> whose uppercased name is a THERMOGRAPH_* var, unless already set in the process env; THERMOGRAPH_DATABASE_URL is built from a postgres_password secret the same way compose's own interpolation builds it today. Verified against fake alembic/uvicorn binaries in a throwaway container: default boot, migrate-only mode (both trigger forms), RUN_MIGRATIONS=0, secrets populating unset vars, an existing env var beating its secret file, and no /run/secrets present at all (today's plain compose path, unaffected). shellcheck clean.
2026-07-21 00:18:05 +00:00
# One-shot migrate mode (`entrypoint.sh migrate`, or THERMOGRAPH_MIGRATE_ONLY=1): run
# the migration and exit — a one-shot task, so it runs once against the DB rather
# than racing per web replica.
Add entrypoint one-shot migrate mode and a Swarm secrets shim (#231) Two changes to deploy/entrypoint.sh, both needed before the app can run under Docker Swarm without re-triggering known incidents: One-shot migrate mode: `entrypoint.sh migrate` (or THERMOGRAPH_MIGRATE_ONLY=1) runs the Alembic migration and exits, for a Swarm one-shot task that brings the schema to head once rather than racing it across every web replica's boot. RUN_MIGRATIONS=0 skips the inline migrate for a deploy that runs the one-shot task separately. Default (unset) keeps today's behavior unchanged: every boot migrates itself before serving. Secrets shim: the app reads config from os.environ everywhere (an unset THERMOGRAPH_AUTH_SECRET falls back to a random per-process value; an unset VAPID key pair is freshly minted) - Swarm `secrets:` mount each secret as a FILE under /run/secrets/ instead, which the app would never see. Before alembic/uvicorn, mechanically export each /run/secrets/<name> whose uppercased name is a THERMOGRAPH_* var, unless already set in the process env; THERMOGRAPH_DATABASE_URL is built from a postgres_password secret the same way compose's own interpolation builds it today. Verified against fake alembic/uvicorn binaries in a throwaway container: default boot, migrate-only mode (both trigger forms), RUN_MIGRATIONS=0, secrets populating unset vars, an existing env var beating its secret file, and no /run/secrets present at all (today's plain compose path, unaffected). shellcheck clean.
2026-07-21 00:18:05 +00:00
if [ "${1:-}" = "migrate" ] || [ "${THERMOGRAPH_MIGRATE_ONLY:-0}" = "1" ]; then
run_migrations
exit 0
fi
# RUN_MIGRATIONS=0 skips the inline migrate below — for a deploy that runs the
# one-shot task above separately and doesn't want every web replica to also race it.
# Default (unset) keeps today's behavior: every boot brings its own schema to head.
if [ "${RUN_MIGRATIONS:-1}" != "0" ]; then
run_migrations
fi
Containerize the app and move the databases to PostgreSQL 18 (#220) Run Thermograph as a docker-compose stack (app + Postgres 18) and standardize the data layer on Postgres, while keeping the test suite on SQLite. - accounts/db.py: DSN-driven engines. On Postgres, a per-worker read-write + read-only asyncpg pair (the RO engine pins read-only transactions, used by the pure-GET endpoints) plus a sync psycopg engine for the notifier thread; the SQLite path is preserved for tests/local (selected when THERMOGRAPH_DATABASE_URL is unset). models.py: boolean server_default -> sa.false(). - store.py / metrics.py: dialect-flexible — Postgres UNLOGGED tables via psycopg when configured, else the existing raw-sqlite3 paths byte-for-byte; sync interfaces and every fail-soft contract preserved. - Alembic (backend/alembic/) manages the accounts schema; the container entrypoint runs `alembic upgrade head` before uvicorn (4 workers). migrate_accounts_to_pg.py copies the accounts data SQLite->PG through the ORM (UUID/bool/JSON coerced), skips access_token, and resets identity sequences. - Dockerfile + docker-compose.yml: app image (uvicorn, 4 workers, loopback 8137) and a Postgres 18 db (2 CPUs) running pg_duckdb (deploy/db/) so the parquet climate cache is queryable in-DB via read_parquet('/parquet/cache/*.parquet'). - deploy.sh/thermograph.service rewired to manage the compose stack; env example, Makefile targets (up/down/db-up), and deploy/POSTGRES-MIGRATION.md cutover runbook. Tests stay on SQLite (dialect fallback) — 323 pass. The full Postgres stack was verified via docker compose: alembic migrations, register/login, the RO endpoint, store/metrics round-trips, and the accounts data migration.
2026-07-20 06:28:23 +00:00
echo "==> Starting uvicorn on 0.0.0.0:${PORT:-8137} with ${WORKERS:-4} worker(s)"
# --no-access-log: the app's own request-logging middleware (audit.log_access,
# web/app.py) already writes a structured line per request; uvicorn's own access
# log just duplicated every one of them for no benefit.
exec uvicorn app:app --host 0.0.0.0 --port "${PORT:-8137}" --workers "${WORKERS:-4}" --no-access-log