083d3bd0f8
All checks were successful
PR build (required check) / changes (pull_request) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Successful in 1m8s
PR build (required check) / build-backend (pull_request) Successful in 1m36s
PR build (required check) / gate (pull_request) Successful in 2s
A visitor who declines browser geolocation currently has no location at all — the copy sends them to the map picker. This adds an opt-in fallback that suggests a coarse city from the request's own IP, presented as a guess with a one-tap correction, so the dead end has a way out. backend/data/geoip.py does the lookup against a local MMDB file (DB-IP IP to City Lite or GeoLite2 City — same format, either works, attribution derived from the file's metadata). Off unless THERMOGRAPH_GEOIP is truthy AND the database exists AND maxminddb imports; every degraded case — private/reserved/ CGNAT/IPv6-ULA addresses, unparseable input, no record, a country-centroid record with no city, a record wider than the accuracy limit, a corrupt file — returns None, which the route answers as 204 and the client treats exactly as today. The IP is read in memory and dropped. GET /api/v2/geoip runs no RunAudit, records no metric dimension, and is excluded from the access log (its own "geoip" traffic category) so no stored, joinable (IP, location) pair is ever written. The response is no-store/Vary:* and takes no parameters. Client-side rather than server-rendered on purpose: the homepage's HTML and weak ETag must stay byte-identical for every visitor, or any cache added in front of it can serve one visitor's city to another. The suggestion is fetched only after a declined/unavailable prompt, and only when nothing is remembered. A guess is never persisted — no localStorage write, no URL hash — and the hero and results headings say "roughly near"/"near … approximate" rather than letting the reverse-geocoded cell name a neighbourhood the lookup never knew. The /privacy page's "never looks up your location from your IP address" paragraph now follows the same flag out of the same env file, so the published statement and the behaviour flip together. Database lifecycle is a host-side systemd timer (infra/deploy/geoip-refresh.*) that verifies a download opens and answers before swapping it in atomically, bind-mounted read-only; geoip.py re-opens on mtime change, so a refresh needs no restart. GEOIP-APPROX-LOCATION.md carries the database comparison, the SSR-vs-client argument, the privacy analysis, and the open decisions.
259 lines
13 KiB
YAML
259 lines
13 KiB
YAML
# Thermograph production stack: backend (FastAPI/API + TestClient(app.py)),
|
|
# frontend (the SSR content service), and TimescaleDB (PostgreSQL 18).
|
|
#
|
|
# FE/BE CI-CD split (finishes repo-split Stage 6/7): backend and frontend are
|
|
# two containers, each running its OWN image published by its OWN repo's
|
|
# build-push.yml -- backend = ${BACKEND_IMAGE_PATH}, frontend =
|
|
# ${FRONTEND_IMAGE_PATH}, tagged independently by BACKEND_IMAGE_TAG /
|
|
# FRONTEND_IMAGE_TAG. This replaces the earlier Stage-4 model where both
|
|
# containers shared the single emi/thermograph/app image and
|
|
# THERMOGRAPH_SERVICE_ROLE picked the process; the split Dockerfiles now start
|
|
# the right process directly (frontend `uvicorn app:app`, backend
|
|
# entrypoint.sh), so a backend deploy and a frontend deploy are fully
|
|
# independent -- deploy.sh rolls one service without touching the other's tag.
|
|
# Both get their own loopback-published port since prod/beta's host Caddy
|
|
# path-splits directly to each; backend also gets a reverse-proxy fallback to
|
|
# frontend (THERMOGRAPH_FRONTEND_BASE_INTERNAL) for any environment with no
|
|
# Caddy in front (LAN dev, bare-metal run.sh) -- see backend/web/app.py's
|
|
# _proxy_to_frontend.
|
|
#
|
|
# docker compose up -d --build # or: make up
|
|
#
|
|
# POSTGRES_PASSWORD must be set at `docker compose` time — compose reads it from
|
|
# the repo-root .env for local runs (copy .env.example -> .env), and in prod the
|
|
# systemd unit's EnvironmentFile=/etc/thermograph.env puts it in the environment
|
|
# so `docker compose up` can interpolate it. It is used BOTH to initialize the db
|
|
# container and to build backend's THERMOGRAPH_DATABASE_URL below.
|
|
|
|
# Pin the compose project name: the monorepo layout runs compose from
|
|
# /opt/thermograph/infra, and without an explicit name the project would be
|
|
# derived from that directory ("infra") -- silently a NEW project, recreating
|
|
# the whole stack beside the running one. LAN dev overrides this via
|
|
# COMPOSE_PROJECT_NAME=thermograph-dev (the env var always wins over this key).
|
|
name: thermograph
|
|
|
|
services:
|
|
db:
|
|
# TimescaleDB on PostgreSQL 18 (the stock image already sets
|
|
# shared_preload_libraries=timescaledb). The app's climate record — the full
|
|
# daily archive and the hourly recent+forecast bundle — lives in hypertables
|
|
# here (see backend/data/climate_store.py), so the DB, not the filesystem, is the
|
|
# source of truth. The init script CREATE EXTENSIONs timescaledb on a fresh volume
|
|
# (Alembic also does, idempotently, at app boot).
|
|
#
|
|
# TIMESCALEDB_TAG defaults to the floating latest-pg18 tag (today's behavior,
|
|
# unchanged) so a plain `docker compose up` keeps working with no setup. Pin it
|
|
# to an exact minor (e.g. 2.17.2-pg18) before any host of this stack could ever
|
|
# replicate with another — a floating tag risks two hosts landing on different
|
|
# extension minors, which blocks a physical replica and risks compressed-chunk
|
|
# corruption on restore. docker-stack.yml (the Swarm interim stack) REQUIRES an
|
|
# exact pin for exactly this reason; use the SAME tag on both once you set one.
|
|
image: timescale/timescaledb:${TIMESCALEDB_TAG:-latest-pg18}
|
|
environment:
|
|
POSTGRES_USER: thermograph
|
|
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}
|
|
POSTGRES_DB: thermograph
|
|
# The init tuning script (deploy/db/init/20-tuning.sh) scales the Postgres +
|
|
# DuckDB memory GUCs from this budget, matching the mem_limit below. One knob
|
|
# per host: Terraform sets DB_MEMORY (prod 16g), local/beta default 8g.
|
|
DB_MEMORY: ${DB_MEMORY:-8g}
|
|
volumes:
|
|
# Mount the volume at the PARENT of the data dir and let the image pick its own
|
|
# PGDATA subdir under it (the timescaledb image defaults to
|
|
# /var/lib/postgresql/data). Pinning PGDATA directly at the mountpoint trips an
|
|
# initdb chmod on some Docker setups; the whole tree still persists this way.
|
|
- pgdata:/var/lib/postgresql
|
|
- ./deploy/db/init:/docker-entrypoint-initdb.d
|
|
healthcheck:
|
|
test: ["CMD-SHELL", "pg_isready -U thermograph -d thermograph"]
|
|
interval: 5s
|
|
timeout: 5s
|
|
retries: 10
|
|
# Give Postgres room to cache + process. mem_limit is the hard ceiling; the actual
|
|
# budget is tuned in deploy/db/init/20-tuning.sh, which scales shared_buffers (25%),
|
|
# effective_cache_size (75%), work_mem and maintenance_work_mem from DB_MEMORY — so
|
|
# raising DB_MEMORY raises both the cap and the tuning together. shm_size backs
|
|
# parallel-query shared memory (the 64MB docker default is
|
|
# too small once shared_buffers/parallelism grow).
|
|
# Sized via env (Terraform sets DB_CPUS/DB_MEMORY per host); defaults match the
|
|
# historical 2 CPU / 8 GB budget so a plain `docker compose up` is unchanged.
|
|
mem_limit: ${DB_MEMORY:-8g}
|
|
shm_size: 1gb
|
|
# Cap the DB at DB_CPUS CPUs. Compose v2 honors the top-level `cpus:`; the
|
|
# deploy.resources block is the Swarm-style equivalent, kept for parity.
|
|
cpus: ${DB_CPUS:-2}
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
cpus: "${DB_CPUS:-2}"
|
|
# Must match mem_limit above (compose rejects distinct values).
|
|
memory: ${DB_MEMORY:-8g}
|
|
restart: unless-stopped
|
|
# No host port on purpose: the app reaches Postgres as db:5432 on the
|
|
# compose network. Nothing outside the stack should touch the database.
|
|
|
|
backend:
|
|
# Backend's OWN image, published by thermograph-backend's build-push.yml.
|
|
# No `build:` here -- infra holds no Dockerfile; each service's Dockerfile
|
|
# lives in its own app repo. deploy.sh sets BACKEND_IMAGE_TAG to the SHA
|
|
# build-push.yml pushed for the backend commit being deployed; local dev
|
|
# builds `emi/thermograph/backend:local` from the backend repo (see
|
|
# infra Makefile) and this pulls/uses it. BACKEND_IMAGE_PATH/TAG are
|
|
# independent of the frontend's, so the two services deploy separately.
|
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:-local}
|
|
depends_on:
|
|
db:
|
|
condition: service_healthy
|
|
environment:
|
|
# Built from POSTGRES_PASSWORD; this `environment` value wins over anything
|
|
# in env_file, so the URL always matches the db container's password.
|
|
THERMOGRAPH_DATABASE_URL: postgresql+asyncpg://thermograph:${POSTGRES_PASSWORD}@db:5432/thermograph
|
|
THERMOGRAPH_BASE: /
|
|
PORT: 8137
|
|
THERMOGRAPH_SERVICE_ROLE: backend
|
|
# Docker's built-in service-name DNS -- reachable only inside the compose
|
|
# network. See backend/web/app.py's _proxy_to_frontend.
|
|
THERMOGRAPH_FRONTEND_BASE_INTERNAL: http://frontend:8080
|
|
# Worker count is env-driven so Terraform can raise it on a bigger host;
|
|
# defaults to 4 to keep a plain `docker compose up` identical to before.
|
|
WORKERS: ${WORKERS:-4}
|
|
# One worker wins this lock and runs the subscription notifier / homepage
|
|
# sweep; it lives on the appdata volume so it's shared across workers.
|
|
THERMOGRAPH_DATA_DIR: /state
|
|
# notifier.lock lives in the state volume, which is now mounted at /state
|
|
# (NOT /app/data) so it never shadows the Python `data/` package -- see the
|
|
# volumes: note below and thermograph-backend paths.py.
|
|
THERMOGRAPH_SINGLETON_LOCK: /state/notifier.lock
|
|
# History reads ask the lake service before any third-party source; if
|
|
# the lake has no bucket creds (LAN without them exported) it answers
|
|
# 503 and climate.py falls straight through to NASA — an accelerator,
|
|
# never a dependency.
|
|
THERMOGRAPH_LAKE_URL: http://lake:8141
|
|
# Approximate-location fallback (backend/data/geoip.py). OFF unless
|
|
# THERMOGRAPH_GEOIP is set truthy in /etc/thermograph.env, and inert even
|
|
# then until deploy/geoip-refresh.sh has landed a database at the host
|
|
# path bind-mounted below — so shipping the code changes nothing by
|
|
# itself, and the mount being empty is a supported steady state.
|
|
THERMOGRAPH_GEOIP: ${THERMOGRAPH_GEOIP:-}
|
|
THERMOGRAPH_GEOIP_DB: /geoip/city.mmdb
|
|
# Prod secrets live in /etc/thermograph.env: POSTGRES_PASSWORD,
|
|
# THERMOGRAPH_AUTH_SECRET, THERMOGRAPH_VAPID_PRIVATE_KEY/_PUBLIC_KEY,
|
|
# THERMOGRAPH_COOKIE_SECURE=1, mail/Discord keys, ... (see
|
|
# deploy/thermograph.env.example). `required: false` so local `docker compose
|
|
# up` works without that file — it reads POSTGRES_PASSWORD from repo-root .env.
|
|
env_file:
|
|
- path: /etc/thermograph.env
|
|
required: false
|
|
volumes:
|
|
# Parquet cache, notifier.lock, homepage.json, vapid.json persist here.
|
|
# /state, NOT /app/data: after the repo split the backend's Python package
|
|
# `data/` sits at /app/data, so mounting the runtime volume there erased
|
|
# data/*.py and broke `import data.climate` at boot. THERMOGRAPH_DATA_DIR=/state
|
|
# (above) points runtime state here instead, clear of the code.
|
|
- appdata:/state
|
|
- applogs:/app/logs
|
|
# The IP-geolocation database, refreshed on the HOST by
|
|
# deploy/geoip-refresh.sh (systemd timer, monthly) and mounted read-only:
|
|
# the app must never be able to write it, and a 100-200 MB data file that
|
|
# goes stale on its own schedule does not belong baked into the image.
|
|
# geoip.py re-opens on mtime change, so a refresh needs no restart.
|
|
- ${THERMOGRAPH_GEOIP_HOST_DIR:-/var/lib/thermograph/geoip}:/geoip:ro
|
|
# No compose-level healthcheck override -- the image's own Dockerfile
|
|
# HEALTHCHECK (port-aware via ${PORT}) already covers this, and frontend's
|
|
# depends_on below reads it the same way.
|
|
# Sized via env (Terraform sets APP_CPUS per host); defaults to 4 CPUs. The
|
|
# deploy.resources block mirrors the top-level `cpus:` for Swarm parity; the
|
|
# dev overlay drops both so dev runs uncapped.
|
|
cpus: ${APP_CPUS:-4}
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
cpus: "${APP_CPUS:-4}"
|
|
ports:
|
|
# Loopback only — host Caddy terminates TLS and reverse-proxies to this
|
|
# (and, in prod/beta, directly to frontend's own port below too).
|
|
- "127.0.0.1:8137:8137"
|
|
restart: unless-stopped
|
|
|
|
# The ERA5 lake service: partition-pruned reads over the era5-thermograph
|
|
# bucket with a local parquet cache, same backend image with
|
|
# THERMOGRAPH_ROLE=lake (the entrypoint starts lake_app — no database, no
|
|
# migrations, so no depends_on: db). Bucket creds come from
|
|
# /etc/thermograph.env where the vault renders them (beta/prod) or from the
|
|
# deploy environment (LAN — deploy-dev.yml injects the Actions S3 secrets);
|
|
# without them the service stays healthy and every read falls through to
|
|
# NASA. Never published on a host port: only the backend talks to it.
|
|
lake:
|
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph/backend}:${BACKEND_IMAGE_TAG:-local}
|
|
environment:
|
|
THERMOGRAPH_ROLE: lake
|
|
PORT: 8141
|
|
THERMOGRAPH_SERVICE_ROLE: backend
|
|
WORKERS: "1"
|
|
THERMOGRAPH_LAKE_CACHE: /state/lake-cache
|
|
THERMOGRAPH_LAKE_S3_ACCESS_KEY: ${THERMOGRAPH_LAKE_S3_ACCESS_KEY:-}
|
|
THERMOGRAPH_LAKE_S3_SECRET_KEY: ${THERMOGRAPH_LAKE_S3_SECRET_KEY:-}
|
|
env_file:
|
|
- path: /etc/thermograph.env
|
|
required: false
|
|
volumes:
|
|
- lakecache:/state
|
|
cpus: ${LAKE_CPUS:-2}
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
cpus: "${LAKE_CPUS:-2}"
|
|
restart: unless-stopped
|
|
|
|
frontend:
|
|
# Frontend's OWN image, published by thermograph-frontend's build-push.yml
|
|
# from its own Dockerfile (which starts `uvicorn app:app` directly -- no
|
|
# THERMOGRAPH_SERVICE_ROLE process-picking anymore). Independent
|
|
# FRONTEND_IMAGE_TAG so a frontend deploy never disturbs the backend's tag.
|
|
image: ${REGISTRY_HOST:-git.thermograph.org}/${FRONTEND_IMAGE_PATH:-emi/thermograph/frontend}:${FRONTEND_IMAGE_TAG:-local}
|
|
# Its own register() fetches the IndexNow key from backend at boot -- must
|
|
# wait for a real, healthy backend, not just a started container.
|
|
depends_on:
|
|
backend:
|
|
condition: service_healthy
|
|
environment:
|
|
THERMOGRAPH_BASE: /
|
|
PORT: 8080
|
|
THERMOGRAPH_SERVICE_ROLE: frontend
|
|
THERMOGRAPH_API_BASE_INTERNAL: http://backend:8137
|
|
# Read for ONE purpose: which paragraph the /privacy page renders about
|
|
# location. Frontend behaviour does not branch on it — the interactive
|
|
# tool simply calls the backend endpoint and gets a 204 when the feature
|
|
# is off. Same value as backend's, from the same /etc/thermograph.env, so
|
|
# the published privacy statement can never drift from what the server
|
|
# actually does.
|
|
THERMOGRAPH_GEOIP: ${THERMOGRAPH_GEOIP:-}
|
|
# No volumes -- stateless, holds no data of its own.
|
|
cpus: ${FRONTEND_CPUS:-2}
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
cpus: "${FRONTEND_CPUS:-2}"
|
|
ports:
|
|
- "127.0.0.1:8080:8080"
|
|
restart: unless-stopped
|
|
|
|
volumes:
|
|
pgdata: {}
|
|
appdata: {}
|
|
applogs: {}
|
|
lakecache: {}
|
|
|
|
networks:
|
|
# Pin the default network's subnet + gateway so the host Postfix can rely on a
|
|
# stable address. The app sends verification email by speaking SMTP to the
|
|
# gateway (172.19.0.1), where the host Postfix listens and relays out (see
|
|
# deploy/provision-mail.sh + THERMOGRAPH_SMTP_HOST in thermograph.env). Without
|
|
# the pin, Docker picks a subnet from its pool and the gateway could move.
|
|
# 172.19.0.0/16 matches what prod and beta are live on today, so applying this
|
|
# is a no-op there.
|
|
default:
|
|
ipam:
|
|
config:
|
|
- subnet: 172.19.0.0/16
|
|
gateway: 172.19.0.1
|