deploy.sh + compose: two independent app images, per-service rolls
backend and frontend now reference their own images (BACKEND_IMAGE_PATH/TAG, FRONTEND_IMAGE_PATH/TAG) instead of the shared emi/thermograph/app. deploy.sh takes SERVICE=backend|frontend|all, rolls just that service with --no-deps, persists each service's live tag in deploy/.image-tags.env so a single-service deploy never disturbs the sibling, and fixes the old IMAGE_TAG:? guard that made every workflow deploy hard-fail. Health-checks per service (8137/8080). Claude-Session: https://claude.ai/code/session_01RdARHDJaYC1wSQRTYM6t3Z
This commit is contained in:
Emi Griffith
parent
0e3196fa05
commit
ce454e0be6
2 changed files with 168 additions and 72 deletions
192
deploy/deploy.sh
192
deploy/deploy.sh
|
|
@ -1,22 +1,45 @@
|
|||
#!/usr/bin/env bash
|
||||
# Pull this INFRA repo's checkout up to date, then roll the docker-compose stack
|
||||
# (backend + frontend + PostgreSQL) onto the app image tagged IMAGE_TAG. Run on
|
||||
# the VPS — the app repo's Forgejo Actions workflow invokes this over SSH (see
|
||||
# .forgejo/workflows/deploy.yml there), and you can run it by hand too.
|
||||
# Pull this INFRA repo's checkout up to date, then roll ONE (or all) of the
|
||||
# docker-compose app services onto its separately-published image. Run on the
|
||||
# VPS — each app repo's Forgejo Actions workflow invokes this over SSH (see
|
||||
# .forgejo/workflows/deploy.yml in thermograph-backend / thermograph-frontend),
|
||||
# and you can run it by hand too.
|
||||
#
|
||||
# ssh deploy@vps 'IMAGE_TAG=sha-<12hex> /opt/thermograph/deploy/deploy.sh'
|
||||
# # roll just the backend onto a specific image:
|
||||
# ssh deploy@vps 'SERVICE=backend BACKEND_IMAGE_TAG=sha-<12hex> /opt/thermograph/deploy/deploy.sh'
|
||||
# # roll just the frontend:
|
||||
# ssh deploy@vps 'SERVICE=frontend FRONTEND_IMAGE_TAG=sha-<12hex> /opt/thermograph/deploy/deploy.sh'
|
||||
# # bring the whole stack up (both tags required):
|
||||
# ssh deploy@vps 'SERVICE=all BACKEND_IMAGE_TAG=sha-<a> FRONTEND_IMAGE_TAG=sha-<b> /opt/thermograph/deploy/deploy.sh'
|
||||
#
|
||||
# This checkout is thermograph-infra, not the app repo: BRANCH is this repo's
|
||||
# FE/BE CI-CD split: backend and frontend are published from separate repos as
|
||||
# separate images (emi/thermograph-backend/app, emi/thermograph-frontend/app),
|
||||
# so a deploy targets ONE service and leaves the other's running container +
|
||||
# tag untouched. Each service's live tag is persisted host-side in
|
||||
# deploy/.image-tags.env (untracked -- survives the git reset below) so a
|
||||
# single-service roll re-renders compose with BOTH services' real tags and
|
||||
# never accidentally recreates or downgrades the sibling.
|
||||
#
|
||||
# This checkout is thermograph-infra, not an app repo: BRANCH is this repo's
|
||||
# branch (compose files, db init, the secrets vault, this script itself);
|
||||
# IMAGE_TAG is the separately-published app image to run (see the IMAGE_TAG
|
||||
# guard below) -- the two are independent and rarely change together.
|
||||
# the *_IMAGE_TAG values are the separately-published app images to run -- the
|
||||
# two axes are independent and rarely change together.
|
||||
set -euo pipefail
|
||||
|
||||
APP_DIR="${APP_DIR:-/opt/thermograph}"
|
||||
BRANCH="${BRANCH:-main}"
|
||||
# Which service this deploy rolls: backend | frontend | all. Defaults to `all`
|
||||
# (a full-stack bring-up) so a by-hand run with both tags still works; the
|
||||
# per-repo deploy.yml workflows always pass an explicit single service.
|
||||
SERVICE="${SERVICE:-all}"
|
||||
HEALTH_PORT="${HEALTH_PORT:-8137}"
|
||||
cd "$APP_DIR"
|
||||
|
||||
case "$SERVICE" in
|
||||
backend|frontend|all) ;;
|
||||
*) echo "!! SERVICE must be backend|frontend|all, got '$SERVICE'" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
# Secrets (POSTGRES_PASSWORD, VAPID keys, AUTH_SECRET, ...) drive compose
|
||||
# interpolation and are also loaded into the backend container via env_file.
|
||||
#
|
||||
|
|
@ -76,31 +99,61 @@ if [ -z "${DEPLOY_SH_REEXECED:-}" ]; then
|
|||
exec "$0" "$@"
|
||||
fi
|
||||
|
||||
# Registry-pull cutover (repo-split Stage 6): pull the app image build-push.yml
|
||||
# (in the APP repo) already built and pushed, instead of building in place. This
|
||||
# checkout is thermograph-infra, not the app repo, so there is no "current commit"
|
||||
# to derive the tag from the way the old same-repo deploy.sh once did -- IMAGE_TAG
|
||||
# must be supplied explicitly (e.g. by the caller exporting
|
||||
# IMAGE_TAG="sha-<12 hex>" matching the app-repo commit to run, or a semver tag).
|
||||
# Registry-pull cutover: pull the image each app repo's build-push.yml already
|
||||
# built and pushed, instead of building in place. This checkout is
|
||||
# thermograph-infra, not an app repo, so there's no "current commit" to derive
|
||||
# a tag from -- the caller (the deploying repo's deploy.yml) exports its own
|
||||
# service's tag (BACKEND_IMAGE_TAG or FRONTEND_IMAGE_TAG = sha-<12 hex> of the
|
||||
# app commit, or a semver tag).
|
||||
REGISTRY_HOST="${REGISTRY_HOST:-git.thermograph.org}"
|
||||
IMAGE_PATH="${IMAGE_PATH:-emi/thermograph/app}"
|
||||
IMAGE_TAG="${IMAGE_TAG:?set IMAGE_TAG=sha-<12-hex-app-commit> (or a semver tag) -- this checkout can't derive it, it isn't the app repo}"
|
||||
export IMAGE_TAG REGISTRY_HOST IMAGE_PATH
|
||||
export REGISTRY_HOST BACKEND_IMAGE_PATH FRONTEND_IMAGE_PATH
|
||||
|
||||
# Load the last-deployed tag for BOTH services first, so a single-service roll
|
||||
# still renders compose with the sibling's real, currently-running tag (never a
|
||||
# bare `local` that would recreate/downgrade it). The incoming env for the
|
||||
# service being deployed then overrides its line below. This file is untracked
|
||||
# (see .gitignore), so `git reset --hard` above leaves it in place.
|
||||
TAGS_FILE="$APP_DIR/deploy/.image-tags.env"
|
||||
if [ -f "$TAGS_FILE" ]; then
|
||||
set -a; . "$TAGS_FILE"; set +a
|
||||
fi
|
||||
|
||||
# Guard: the service(s) being rolled MUST have a concrete tag supplied now (the
|
||||
# sibling's may come from the persisted file). `all` needs both.
|
||||
case "$SERVICE" in
|
||||
backend) : "${BACKEND_IMAGE_TAG:?set BACKEND_IMAGE_TAG=sha-<12-hex> for a backend deploy}" ;;
|
||||
frontend) : "${FRONTEND_IMAGE_TAG:?set FRONTEND_IMAGE_TAG=sha-<12-hex> for a frontend deploy}" ;;
|
||||
all)
|
||||
: "${BACKEND_IMAGE_TAG:?set BACKEND_IMAGE_TAG=sha-<12-hex> (SERVICE=all needs both)}"
|
||||
: "${FRONTEND_IMAGE_TAG:?set FRONTEND_IMAGE_TAG=sha-<12-hex> (SERVICE=all needs both)}"
|
||||
;;
|
||||
esac
|
||||
# Compose interpolates both vars for the whole file even when we act on one
|
||||
# service; default the not-yet-known sibling (first-ever deploy) to `local` so
|
||||
# interpolation doesn't warn -- harmless since --no-deps never touches it.
|
||||
export BACKEND_IMAGE_TAG="${BACKEND_IMAGE_TAG:-local}"
|
||||
export FRONTEND_IMAGE_TAG="${FRONTEND_IMAGE_TAG:-local}"
|
||||
|
||||
# Which compose services this run pulls/rolls.
|
||||
case "$SERVICE" in
|
||||
backend) TARGETS=(backend) ;;
|
||||
frontend) TARGETS=(frontend) ;;
|
||||
all) TARGETS=(backend frontend) ;;
|
||||
esac
|
||||
|
||||
echo "==> Logging in to the registry ($REGISTRY_HOST)"
|
||||
echo "$REGISTRY_TOKEN" | docker login "$REGISTRY_HOST" --username emi --password-stdin
|
||||
|
||||
echo "==> Pulling images ($IMAGE_TAG)"
|
||||
# Retry: build-push.yml (triggered by the same push) has no ordering
|
||||
# guarantee against this deploy -- Forgejo/GitHub Actions `needs:` only
|
||||
# works between jobs in ONE workflow file, not across two workflows
|
||||
# triggered by the same event. Confirmed live: a real deploy raced ahead of
|
||||
# the push and failed with "not found". A bounded retry (~5 min) comfortably
|
||||
# covers a normal build; a genuine problem (bad tag, registry down) still
|
||||
# fails loudly after that, just a bit slower to surface.
|
||||
echo "==> Pulling images (backend=$BACKEND_IMAGE_TAG frontend=$FRONTEND_IMAGE_TAG; rolling: ${TARGETS[*]})"
|
||||
# Retry: build-push.yml (triggered by the same push) has no ordering guarantee
|
||||
# against this deploy -- Forgejo Actions `needs:` only works between jobs in ONE
|
||||
# workflow file, not across the separate build-push.yml triggered by the same
|
||||
# event. Confirmed live: a deploy raced ahead of the push and failed with "not
|
||||
# found". A bounded retry (~5 min) covers a normal build; a genuine problem
|
||||
# (bad tag, registry down) still fails loudly after that.
|
||||
pull_ok=0
|
||||
for i in $(seq 1 30); do
|
||||
if docker compose pull backend frontend; then
|
||||
if docker compose pull "${TARGETS[@]}"; then
|
||||
pull_ok=1
|
||||
break
|
||||
fi
|
||||
|
|
@ -112,32 +165,67 @@ if [ "$pull_ok" != 1 ]; then
|
|||
exit 1
|
||||
fi
|
||||
|
||||
# Schema migrations run inside the backend container's entrypoint (alembic
|
||||
# upgrade head) before uvicorn starts, so there's no separate migrate step or
|
||||
# service restart here — compose owns the process model. frontend has no
|
||||
# migrations (stateless). --remove-orphans matters whenever the compose
|
||||
# service topology itself changes (e.g. the old single `app` service ->
|
||||
# `backend`+`frontend`, repo-split Stage 4): compose only manages containers
|
||||
# for services CURRENTLY defined in the file, so a plain `up -d` leaves a
|
||||
# renamed-away service's old container running and squatting its port
|
||||
# forever -- confirmed live, this is exactly what blocked beta's first
|
||||
# dual-service deploy ("port is already allocated").
|
||||
echo "==> Starting stack"
|
||||
docker compose up -d --remove-orphans
|
||||
# Roll only the target service(s). Backend schema migrations run inside its own
|
||||
# entrypoint (alembic upgrade head) before uvicorn, so there's no separate
|
||||
# migrate step; frontend is stateless.
|
||||
#
|
||||
# Single-service rolls use --no-deps so recreating backend doesn't also bounce
|
||||
# db, and recreating frontend doesn't touch backend -- that independence is the
|
||||
# whole point of the split. A full `all` deploy instead uses --remove-orphans,
|
||||
# which matters when the service topology itself changes (a renamed-away
|
||||
# service's old container would otherwise keep running and squat its port --
|
||||
# confirmed live, this is what blocked beta's first dual-service deploy with
|
||||
# "port is already allocated").
|
||||
echo "==> Rolling ${TARGETS[*]}"
|
||||
if [ "$SERVICE" = all ]; then
|
||||
docker compose up -d --remove-orphans
|
||||
else
|
||||
docker compose up -d --no-deps "${TARGETS[@]}"
|
||||
fi
|
||||
|
||||
echo "==> Health check"
|
||||
url="http://127.0.0.1:${HEALTH_PORT}/"
|
||||
for i in $(seq 1 30); do
|
||||
if curl -fsS -o /dev/null "$url"; then
|
||||
echo "==> OK: $url is serving"
|
||||
warm_city_archives
|
||||
ping_indexnow
|
||||
exit 0
|
||||
# Persist the now-live tags so the next single-service deploy knows the
|
||||
# sibling's real tag. Written after `up` so a failed pull never records a tag
|
||||
# that isn't actually running.
|
||||
mkdir -p "$(dirname "$TAGS_FILE")"
|
||||
cat > "$TAGS_FILE" <<EOF
|
||||
# Written by deploy.sh -- the image tag each service is currently running.
|
||||
# Untracked (see .gitignore); lets a single-service deploy leave the other alone.
|
||||
BACKEND_IMAGE_TAG=$BACKEND_IMAGE_TAG
|
||||
FRONTEND_IMAGE_TAG=$FRONTEND_IMAGE_TAG
|
||||
EOF
|
||||
|
||||
# Health check the service(s) we rolled: backend on 8137, frontend on 8080.
|
||||
# (For `all`, backend's `/` serving is the readiness signal the old script used
|
||||
# and the frontend depends_on backend anyway.)
|
||||
declare -A HEALTH_PORTS=( [backend]=8137 [frontend]=8080 )
|
||||
health_ok=1
|
||||
for svc in "${TARGETS[@]}"; do
|
||||
port="${HEALTH_PORTS[$svc]}"
|
||||
url="http://127.0.0.1:${port}/healthz"
|
||||
echo "==> Health check: $svc ($url)"
|
||||
ok=0
|
||||
for i in $(seq 1 30); do
|
||||
if curl -fsS -o /dev/null "$url"; then ok=1; break; fi
|
||||
sleep 1
|
||||
done
|
||||
if [ "$ok" = 1 ]; then
|
||||
echo "==> OK: $svc is serving"
|
||||
else
|
||||
echo "!! Health check failed for $svc ($url)" >&2
|
||||
health_ok=0
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
echo "!! Health check failed for $url" >&2
|
||||
docker compose ps || true
|
||||
docker compose logs --tail=50 backend || true
|
||||
docker compose logs --tail=50 frontend || true
|
||||
exit 1
|
||||
|
||||
if [ "$health_ok" != 1 ]; then
|
||||
docker compose ps || true
|
||||
for svc in "${TARGETS[@]}"; do docker compose logs --tail=50 "$svc" || true; done
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Post-deploy warm/IndexNow only make sense once the backend is (re)deployed --
|
||||
# they exec inside the backend container. Skip them on a frontend-only roll.
|
||||
if [ "$SERVICE" = backend ] || [ "$SERVICE" = all ]; then
|
||||
warm_city_archives
|
||||
ping_indexnow
|
||||
fi
|
||||
exit 0
|
||||
|
|
|
|||
|
|
@ -1,14 +1,21 @@
|
|||
# Thermograph production stack: backend (FastAPI/API + TestClient(app.py)),
|
||||
# frontend (the SSR content service), and TimescaleDB (PostgreSQL 18).
|
||||
#
|
||||
# Repo-split Stage 4: backend and frontend are two containers built from the
|
||||
# SAME image (THERMOGRAPH_SERVICE_ROLE picks which process runs -- see
|
||||
# Dockerfile/deploy/entrypoint.sh), not one "app" service anymore. Both get
|
||||
# their own loopback-published port since prod/beta's host Caddy path-splits
|
||||
# directly to each; backend also gets a reverse-proxy fallback to frontend
|
||||
# (THERMOGRAPH_FRONTEND_BASE_INTERNAL) for any environment with no Caddy in
|
||||
# front (LAN dev, bare-metal run.sh) -- see backend/web/app.py's
|
||||
# _proxy_to_frontend and the repo-split plan's Stage 4 section.
|
||||
# FE/BE CI-CD split (finishes repo-split Stage 6/7): backend and frontend are
|
||||
# two containers, each running its OWN image published by its OWN repo's
|
||||
# build-push.yml -- backend = ${BACKEND_IMAGE_PATH}, frontend =
|
||||
# ${FRONTEND_IMAGE_PATH}, tagged independently by BACKEND_IMAGE_TAG /
|
||||
# FRONTEND_IMAGE_TAG. This replaces the earlier Stage-4 model where both
|
||||
# containers shared the single emi/thermograph/app image and
|
||||
# THERMOGRAPH_SERVICE_ROLE picked the process; the split Dockerfiles now start
|
||||
# the right process directly (frontend `uvicorn app:app`, backend
|
||||
# entrypoint.sh), so a backend deploy and a frontend deploy are fully
|
||||
# independent -- deploy.sh rolls one service without touching the other's tag.
|
||||
# Both get their own loopback-published port since prod/beta's host Caddy
|
||||
# path-splits directly to each; backend also gets a reverse-proxy fallback to
|
||||
# frontend (THERMOGRAPH_FRONTEND_BASE_INTERNAL) for any environment with no
|
||||
# Caddy in front (LAN dev, bare-metal run.sh) -- see backend/web/app.py's
|
||||
# _proxy_to_frontend.
|
||||
#
|
||||
# docker compose up -d --build # or: make up
|
||||
#
|
||||
|
|
@ -79,14 +86,14 @@ services:
|
|||
# compose network. Nothing outside the stack should touch the database.
|
||||
|
||||
backend:
|
||||
build: .
|
||||
# Named so `docker compose pull` can fetch a prebuilt tag instead of
|
||||
# `--build`ing in place (repo-split Stage 6's registry-pull cutover --
|
||||
# deploy.sh/Terraform set IMAGE_TAG to the SHA build-push.yml just pushed
|
||||
# for the branch being deployed). Harmless default for local/CI
|
||||
# `--build` usage, which never needs to resolve this against the
|
||||
# registry -- `build:` + `image:` together just tag the local build.
|
||||
image: ${REGISTRY_HOST:-git.thermograph.org}/${IMAGE_PATH:-emi/thermograph/app}:${IMAGE_TAG:-local}
|
||||
# Backend's OWN image, published by thermograph-backend's build-push.yml.
|
||||
# No `build:` here -- infra holds no Dockerfile; each service's Dockerfile
|
||||
# lives in its own app repo. deploy.sh sets BACKEND_IMAGE_TAG to the SHA
|
||||
# build-push.yml pushed for the backend commit being deployed; local dev
|
||||
# builds `emi/thermograph-backend/app:local` from the backend repo (see
|
||||
# infra Makefile) and this pulls/uses it. BACKEND_IMAGE_PATH/TAG are
|
||||
# independent of the frontend's, so the two services deploy separately.
|
||||
image: ${REGISTRY_HOST:-git.thermograph.org}/${BACKEND_IMAGE_PATH:-emi/thermograph-backend/app}:${BACKEND_IMAGE_TAG:-local}
|
||||
depends_on:
|
||||
db:
|
||||
condition: service_healthy
|
||||
|
|
@ -136,10 +143,11 @@ services:
|
|||
restart: unless-stopped
|
||||
|
||||
frontend:
|
||||
build: .
|
||||
# Same image as backend (see backend's `image:` comment above) --
|
||||
# THERMOGRAPH_SERVICE_ROLE below is what picks the process.
|
||||
image: ${REGISTRY_HOST:-git.thermograph.org}/${IMAGE_PATH:-emi/thermograph/app}:${IMAGE_TAG:-local}
|
||||
# Frontend's OWN image, published by thermograph-frontend's build-push.yml
|
||||
# from its own Dockerfile (which starts `uvicorn app:app` directly -- no
|
||||
# THERMOGRAPH_SERVICE_ROLE process-picking anymore). Independent
|
||||
# FRONTEND_IMAGE_TAG so a frontend deploy never disturbs the backend's tag.
|
||||
image: ${REGISTRY_HOST:-git.thermograph.org}/${FRONTEND_IMAGE_PATH:-emi/thermograph-frontend/app}:${FRONTEND_IMAGE_TAG:-local}
|
||||
# Its own register() fetches the IndexNow key from backend at boot -- must
|
||||
# wait for a real, healthy backend, not just a started container.
|
||||
depends_on:
|
||||
|
|
|
|||
Loading…
Reference in a new issue