From 8a2c838663a1012996ac35121c48c6dfd8a3632b Mon Sep 17 00:00:00 2001 From: Emi Griffith Date: Wed, 29 Jul 2026 21:42:01 -0700 Subject: [PATCH] forgejo: restart the db service on a clean exit, not just on failure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit forgejo_db has been at 0/1 replicas since 2026-07-29 01:24 UTC. Postgres hit an invalid data-directory lock file ("could not open file postmaster.pid ... performing immediate shutdown because data directory lock file is invalid") and exited 0. With restart_policy.condition=on-failure, Swarm read the zero status as successful completion, marked the task Complete, and never rescheduled it. The forgejo service itself stayed Up and kept serving its homepage, so the outage presented as every repository page, the whole API and all CI returning 500 with "dial tcp: lookup db on 127.0.0.11:53: no such host" — including the auth path, which is why API calls reported "user does not exist [uid: 0]" rather than a database error. on-failure cannot distinguish "finished successfully" from "shut itself down and should be restarted", and Postgres exits 0 on several such paths, so it is the wrong policy for an always-on stateful service. This is the durable fix; it does not restart the currently stopped task. --- infra/deploy/forgejo/docker-stack.yml | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/infra/deploy/forgejo/docker-stack.yml b/infra/deploy/forgejo/docker-stack.yml index a86d41b..b939025 100644 --- a/infra/deploy/forgejo/docker-stack.yml +++ b/infra/deploy/forgejo/docker-stack.yml @@ -53,7 +53,19 @@ services: cpus: "${FORGEJO_DB_CPUS:-1}" memory: ${FORGEJO_DB_MEMORY:-1g} restart_policy: - condition: on-failure + # `any`, NOT `on-failure`. This took Forgejo down for 27 hours on 2026-07-29: + # Postgres hit an invalid data-directory lock file ("could not open file + # postmaster.pid ... performing immediate shutdown") and exited **0**. A clean + # exit is not a failure, so Swarm considered the task Complete, dropped the + # service to 0/1 replicas, and never rescheduled it. Forgejo itself stayed Up + # and served its homepage while every repo page, the API and all CI returned + # 500 with `dial tcp: lookup db ... no such host`. + # + # `on-failure` is the wrong policy for any always-on stateful service: it + # cannot distinguish "finished successfully" from "shut itself down and should + # be restarted", and Postgres does the latter with status 0 on several paths. + condition: any + delay: 5s forgejo: image: codeberg.org/forgejo/forgejo:9-rootless