6086b40fe6
All checks were successful
secrets-guard / encrypted (pull_request) Successful in 7s
PR build (required check) / changes (pull_request) Successful in 8s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / gate (pull_request) Successful in 3s
Prod DB dumps and Forgejo (db + data volume) are now copied off-box to the era5-thermograph S3 bucket, age-encrypted to the vault recipient so the host age key at /etc/thermograph/age.key decrypts them for restore. Nothing plaintext leaves the box; no local intermediate (streamed via age | rclone rcat). ops-cron: the prod pg_dump job gains an off-box push; a new forgejo-backup job dumps Forgejo's Postgres + tars its data volume on beta (the git host had NO backup at all). 30-day off-box retention on both. S3 creds are Forgejo Actions secrets (S3_ENDPOINT/BUCKET/ACCESS_KEY/SECRET_KEY), mirrored into the SOPS vault (prod.yaml/beta.yaml) for host-side use. Contabo needs path-style addressing. Adds deploy/backup/README.md with the DR restore runbook.
66 lines
3 KiB
Markdown
66 lines
3 KiB
Markdown
# Off-box backups → Contabo Object Storage
|
|
|
|
Durable, encrypted, off-box copies of the state that would otherwise die with a
|
|
single VPS. Driven by `.forgejo/workflows/ops-cron.yml` (03:00 UTC daily +
|
|
`workflow_dispatch`).
|
|
|
|
## What is backed up
|
|
|
|
| Prefix in bucket `era5-thermograph` | Source | Job | Retention off-box |
|
|
|---|---|---|---|
|
|
| `backups/db/prod/` | prod Postgres/TimescaleDB (`pg_dump --format=custom`) | `backup` (prod, `PROD_SSH_*`) | 30 days |
|
|
| `backups/forgejo/db/` | Forgejo Postgres (`forgejo_db`) | `forgejo-backup` (beta, `SSH_*`) | 30 days |
|
|
| `backups/forgejo/data/` | Forgejo data volume (repos/LFS/config) | `forgejo-backup` | 30 days |
|
|
|
|
Everything is streamed through **`age`** encryption before upload — encrypted to the
|
|
vault recipient `age1xx4dz…`, so the private key each host already has at
|
|
`/etc/thermograph/age.key` (and the operator's `~/.config/sops/age/keys.txt`)
|
|
decrypts it. Nothing plaintext leaves the box.
|
|
|
|
## Access layer
|
|
- S3 endpoint/bucket/keys are **Forgejo Actions secrets** (`S3_ENDPOINT`,
|
|
`S3_BUCKET`, `S3_ACCESS_KEY`, `S3_SECRET_KEY`) — the CI job's credential home
|
|
(also mirrored into the SOPS vault `deploy/secrets/{prod,beta}.yaml` for host-side
|
|
use). Contabo needs **path-style** addressing (`force_path_style=true`,
|
|
`provider=Other`, `region=default`).
|
|
- `rclone` + `age` must be installed on prod and beta (one-time: `apt-get install -y
|
|
rclone age`). The prod job apt-installs them if missing; the beta user has no sudo,
|
|
so they are pre-installed there.
|
|
|
|
## Restore (DR runbook)
|
|
|
|
Prereq on the restoring host: `rclone` + `age` + the age key at
|
|
`/etc/thermograph/age.key` (or the operator key), and rclone configured for the
|
|
`archive:` remote (env vars `RCLONE_CONFIG_ARCHIVE_*` or `/etc/rclone/rclone.conf`).
|
|
|
|
### Prod database
|
|
```sh
|
|
# list available encrypted dumps
|
|
rclone ls archive:era5-thermograph/backups/db/prod/
|
|
# download newest, decrypt, restore into a fresh db
|
|
OBJ=archive:era5-thermograph/backups/db/prod/<name>.dump.age
|
|
rclone cat "$OBJ" | sudo age -d -i /etc/thermograph/age.key \
|
|
| docker exec -i <db-container> pg_restore -U thermograph -d thermograph --clean --if-exists
|
|
```
|
|
|
|
### Forgejo
|
|
```sh
|
|
# database
|
|
rclone cat archive:era5-thermograph/backups/forgejo/db/<name>.dump.age \
|
|
| sudo age -d -i /etc/thermograph/age.key \
|
|
| docker exec -i forgejo_db sh -c 'PGPASSWORD=$POSTGRES_PASSWORD pg_restore -U $POSTGRES_USER -d $POSTGRES_DB --clean --if-exists'
|
|
# data volume (repos/lfs/config) — restore into the volume with Forgejo stopped
|
|
rclone cat archive:era5-thermograph/backups/forgejo/data/<name>.tar.age \
|
|
| sudo age -d -i /etc/thermograph/age.key \
|
|
| docker run --rm -i -v forgejo_forgejo_data:/data busybox tar -C /data -xf -
|
|
```
|
|
|
|
## Verify a backup is restorable (quick)
|
|
```sh
|
|
rclone cat archive:era5-thermograph/backups/db/prod/<name>.dump.age \
|
|
| sudo age -d -i /etc/thermograph/age.key | head -c 5 # -> PGDMP = valid custom-format dump
|
|
```
|
|
|
|
## Manual run
|
|
`workflow_dispatch` the "Ops cron (backup + IndexNow)" workflow, or run the same
|
|
`age | rclone rcat` pipeline by hand (see the workflow script).
|