forgejo: restart the db service on a clean exit, not just on failure #128

Merged

admin_emi merged 1 commit from forgejo-db-restart-policy into dev

2026-07-30 18:09:21 +00:00

admin_emi commented

2026-07-30 13:30:53 +00:00

Owner

What happened

forgejo_db sat at 0/1 replicas from 2026-07-29 01:24 UTC until 2026-07-30, roughly 27 hours.

Postgres hit an invalid data-directory lock file:

LOG:  could not open file "postmaster.pid": No such file or directory
LOG:  performing immediate shutdown because data directory lock file is invalid
LOG:  database system is shut down

…and exited 0. With restart_policy.condition: on-failure, Swarm read that zero status as successful completion, marked the task Complete, dropped the service to 0 replicas, and never rescheduled it.

Why it was hard to spot

The forgejo service itself stayed 1/1 Up and kept serving its homepage, so / returned 200 the whole time. Everything needing the database returned 500:

every repository page, and the whole git-over-HTTP path (so no clones, no pushes)
the entire API
all CI, with dial tcp: lookup db on 127.0.0.11:53: no such host

The auth path failed the same way, which made API calls report 401 {"message":"user does not exist [uid: 0, name: ]"} — Forgejo could not read its own user table. That misleadingly looks like an expired token, and was initially diagnosed as one. CENTRALIS_FORGEJO_TOKEN was never expired and needed no rotation.

The fix

condition: any with a 5s delay.

on-failure cannot distinguish "finished successfully" from "shut itself down and should be restarted", and Postgres exits 0 on several such paths. It is the wrong policy for any always-on stateful service.

This is the durable fix only — it prevents recurrence but does not itself recover a database. Recovery was separate: the data directory's WAL was unrecoverable (invalid contrecord length, invalid checkpoint record, PANIC: could not locate a valid checkpoint record), so the service was restored from the 2026-07-28T03:00Z off-box dump. The corrupt data directory is preserved byte-for-byte in the forgejo_db_corrupt_20260730 volume on vps1, with pg_control and pg_wal intact, in case a pg_resetwal attempt to recover the intervening ~22 hours of metadata is ever wanted.

Follow-ups this exposed, not addressed here

The same on-failure policy should be audited on every other stateful service in the estate's stack files.
Nothing alerted on this for 27 hours. A 0-replica Swarm service on any stack is worth a Grafana alert; the homepage staying 200 is exactly why a naive uptime check missed it.
Backups silently stopped too. The forgejo-backup job needs a live forgejo_db to pg_dump, so there are no backups/forgejo/db/ objects dated 07-29 or 07-30. A backup job that cannot find its source should fail loudly rather than leave a gap.

## What happened `forgejo_db` sat at **0/1 replicas from 2026-07-29 01:24 UTC until 2026-07-30**, roughly 27 hours. Postgres hit an invalid data-directory lock file: ``` LOG: could not open file "postmaster.pid": No such file or directory LOG: performing immediate shutdown because data directory lock file is invalid LOG: database system is shut down ``` …and exited **0**. With `restart_policy.condition: on-failure`, Swarm read that zero status as successful completion, marked the task `Complete`, dropped the service to 0 replicas, and never rescheduled it. ## Why it was hard to spot The `forgejo` service itself stayed `1/1 Up` and kept serving its homepage, so `/` returned 200 the whole time. Everything needing the database returned 500: - every repository page, and the whole git-over-HTTP path (so no clones, no pushes) - the entire API - all CI, with `dial tcp: lookup db on 127.0.0.11:53: no such host` The auth path failed the same way, which made API calls report `401 {"message":"user does not exist [uid: 0, name: ]"}` — Forgejo could not read its own user table. That misleadingly looks like an expired token, and was initially diagnosed as one. `CENTRALIS_FORGEJO_TOKEN` was never expired and needed no rotation. ## The fix `condition: any` with a 5s delay. `on-failure` cannot distinguish "finished successfully" from "shut itself down and should be restarted", and Postgres exits 0 on several such paths. It is the wrong policy for any always-on stateful service. This is the durable fix only — it prevents recurrence but does not itself recover a database. Recovery was separate: the data directory's WAL was unrecoverable (`invalid contrecord length`, `invalid checkpoint record`, `PANIC: could not locate a valid checkpoint record`), so the service was restored from the `2026-07-28T03:00Z` off-box dump. The corrupt data directory is preserved byte-for-byte in the `forgejo_db_corrupt_20260730` volume on vps1, with `pg_control` and `pg_wal` intact, in case a `pg_resetwal` attempt to recover the intervening ~22 hours of metadata is ever wanted. ## Follow-ups this exposed, not addressed here - **The same `on-failure` policy should be audited on every other stateful service** in the estate's stack files. - **Nothing alerted on this for 27 hours.** A 0-replica Swarm service on any stack is worth a Grafana alert; the homepage staying 200 is exactly why a naive uptime check missed it. - **Backups silently stopped too.** The `forgejo-backup` job needs a live `forgejo_db` to `pg_dump`, so there are no `backups/forgejo/db/` objects dated 07-29 or 07-30. A backup job that cannot find its source should fail loudly rather than leave a gap.

admin_emi added 1 commit 2026-07-30 13:30:54 +00:00

forgejo: restart the db service on a clean exit, not just on failure

shell-lint / shellcheck (pull_request) Successful in 6s

Details

PR build (required check) / gate (pull_request) Successful in 1s

Details

secrets-guard / encrypted (pull_request) Successful in 4s

Details

PR build (required check) / changes (pull_request) Successful in 8s

Details

PR build (required check) / build-backend (pull_request) Has been skipped

Details

PR build (required check) / build-frontend (pull_request) Has been skipped

Details

PR build (required check) / validate-observability (pull_request) Has been skipped

Details

Sync infra to hosts / sync-beta (push) Has been skipped

Details

Sync infra to hosts / sync-prod (push) Has been skipped

Details

Sync infra to hosts / sync-dev (push) Successful in 5s

Details

secrets-guard / encrypted (push) Successful in 4s

Details

shell-lint / shellcheck (push) Successful in 6s

Details

8a2c838663

forgejo_db has been at 0/1 replicas since 2026-07-29 01:24 UTC. Postgres hit
an invalid data-directory lock file ("could not open file postmaster.pid ...
performing immediate shutdown because data directory lock file is invalid")
and exited 0. With restart_policy.condition=on-failure, Swarm read the zero
status as successful completion, marked the task Complete, and never
rescheduled it.

The forgejo service itself stayed Up and kept serving its homepage, so the
outage presented as every repository page, the whole API and all CI returning
500 with "dial tcp: lookup db on 127.0.0.11:53: no such host" — including the
auth path, which is why API calls reported "user does not exist [uid: 0]"
rather than a database error.

on-failure cannot distinguish "finished successfully" from "shut itself down
and should be restarted", and Postgres exits 0 on several such paths, so it is
the wrong policy for an always-on stateful service.

This is the durable fix; it does not restart the currently stopped task.