deploy.sh: fix the daemon-binary probe, which always dropped daemon #33

Merged

admin_emi merged 1 commit from fix-daemon-probe into main

2026-07-24 04:03:27 +00:00

admin_emi commented

2026-07-24 04:02:47 +00:00

Owner

Found while checking live status on beta: thermograph-daemon-1 was still
running an old backend image tag while backend/lake had rolled forward twice
in a row.

Root cause, reproduced directly against beta

docker compose config --images daemon does not filter to the named
service on this host's Compose v5.3.1 — it prints every service's image, one
per line, in file order:

timescale/timescaledb:latest-pg18
git.thermograph.org/emi/thermograph/backend:sha-eaaa4f1ad930
git.thermograph.org/emi/thermograph/backend:sha-eaaa4f1ad930

| head -1 was silently grabbing db's image instead of daemon's. The
probe then always found no /usr/local/bin/thermograph-daemon in a Postgres
image and concluded "this image predates the daemon binary," dropping daemon
from every single backend deploy, regardless of what the real backend
image actually contained — the exact /internal/* version-skew this guard
was written to prevent, caused by the guard itself.

(Prod wasn't affected — it deploys via the Swarm stack path, which reconciles
the full declarative service list on every run rather than rolling a
TARGETS subset, so it never had this failure mode.)

Fix

Build the image reference directly from the same vars docker-compose.yml's
daemon.image: already interpolates
(REGISTRY_HOST/BACKEND_IMAGE_PATH/BACKEND_IMAGE_TAG) instead of going
through docker compose config at all — no dependency on that command's
filtering behavior, and it can't disagree with what compose will actually
run.

Verification

Reproduced the old code's exact command sequence on beta with real
secrets sourced: confirmed it selects the wrong (db) image.
Confirmed the new construction resolves to the correct backend image
and the binary probe passes.
shellcheck -x: 0 findings.
bash -n: clean.

Once this merges I'll trigger a backend redeploy to bring beta's daemon back
in sync (currently 2 releases behind, functionally inert there since the
Discord token is prod-only, but worth correcting).

Found while checking live status on beta: `thermograph-daemon-1` was still running an old backend image tag while backend/lake had rolled forward twice in a row. ## Root cause, reproduced directly against beta `docker compose config --images daemon` does **not** filter to the named service on this host's Compose v5.3.1 — it prints every service's image, one per line, in file order: ``` timescale/timescaledb:latest-pg18 git.thermograph.org/emi/thermograph/backend:sha-eaaa4f1ad930 git.thermograph.org/emi/thermograph/backend:sha-eaaa4f1ad930 ``` `| head -1` was silently grabbing **db's image** instead of daemon's. The probe then always found no `/usr/local/bin/thermograph-daemon` in a Postgres image and concluded "this image predates the daemon binary," dropping daemon from **every single backend deploy**, regardless of what the real backend image actually contained — the exact `/internal/*` version-skew this guard was written to prevent, caused by the guard itself. (Prod wasn't affected — it deploys via the Swarm stack path, which reconciles the full declarative service list on every run rather than rolling a `TARGETS` subset, so it never had this failure mode.) ## Fix Build the image reference directly from the same vars `docker-compose.yml`'s `daemon.image:` already interpolates (`REGISTRY_HOST`/`BACKEND_IMAGE_PATH`/`BACKEND_IMAGE_TAG`) instead of going through `docker compose config` at all — no dependency on that command's filtering behavior, and it can't disagree with what compose will actually run. ## Verification - Reproduced the **old** code's exact command sequence on beta with real secrets sourced: confirmed it selects the wrong (db) image. - Confirmed the **new** construction resolves to the correct backend image and the binary probe passes. - `shellcheck -x`: 0 findings. - `bash -n`: clean. Once this merges I'll trigger a backend redeploy to bring beta's daemon back in sync (currently 2 releases behind, functionally inert there since the Discord token is prod-only, but worth correcting).

admin_emi added 1 commit 2026-07-24 04:02:48 +00:00

deploy.sh: fix the daemon-binary probe, which always dropped daemon

PR build (required check) / changes (pull_request) Successful in 6s

Details

secrets-guard / encrypted (pull_request) Successful in 6s

Details

PR build (required check) / build-backend (pull_request) Has been skipped

Details

PR build (required check) / build-frontend (pull_request) Has been skipped

Details

shell-lint / shellcheck (pull_request) Successful in 8s

Details

PR build (required check) / validate-observability (pull_request) Has been skipped

Details

PR build (required check) / gate (pull_request) Successful in 2s

Details

cf0fa01892

Found while checking live status: beta's thermograph-daemon container was
still running an old backend image tag while backend/lake had rolled
forward twice. Reproduced directly against the host rather than guessing --
`docker compose config --images daemon` does NOT filter to the named
service on this host's Compose v5.3.1: it prints every service's image,
one per line, in file order, so `| head -1` was silently grabbing db's
image (timescaledb) instead of daemon's. The probe then always found no
/usr/local/bin/thermograph-daemon in a Postgres image and concluded "this
image predates the daemon binary," dropping daemon from every single
backend deploy regardless of what the real backend image actually
contained -- the exact /internal/* version-skew this guard was written to
prevent, caused by the guard itself.

Fixed by building the image reference directly from the same vars
docker-compose.yml's daemon.image: already interpolates
(REGISTRY_HOST/BACKEND_IMAGE_PATH/BACKEND_IMAGE_TAG) instead of going
through `docker compose config` at all -- no dependency on that command's
filtering behavior, and it can't disagree with what compose will actually
run.

Verified directly against beta: the old code's exact command sequence
reproduced with real secrets sourced, confirmed the wrong (db) image was
selected; the new construction resolves to the correct backend image and
the binary probe passes.