Jinemi/thermograph
4
0
Fork 0
Code Issues 2 Pull requests 9 Projects Releases Packages Wiki Activity Actions

Off-box encrypted backups to Contabo Object Storage #14

Merged
admin_emi merged 2 commits from feature/storage-backups into dev 2026-07-24 20:08:14 +00:00
Conversation 0 Commits 2 Files changed 4 +219 -62
admin_emi commented 2026-07-23 20:42:37 +00:00
Owner
Copy link

Delivers the storage-integration plan's backup tier. Already live (not just proposed): first backups are in the bucket and restore-verified.

Done manually + verified this session

  • Prod DB: 3 dumps age-encrypted to backups/db/prod/ (~780 MB each). Restore-verified (download -> age-decrypt with host key -> valid PGDMP).
  • Forgejo (previously ZERO backups): db + data volume to backups/forgejo/ (encrypted).
  • rclone + age installed on prod & beta; Contabo creds validated (path-style, region=default).

This PR makes it recurring

  • ops-cron.yml: the prod pg_dump job now age-encrypts + pushes off-box; a new forgejo-backup job backs up Forgejo's db + data volume on beta. 30-day off-box retention on both. Streamed (age | rclone rcat) — no local intermediate, nothing plaintext leaves the box.
  • Encrypted to the vault age recipient, so /etc/thermograph/age.key on each host decrypts for restore.
  • S3 creds are Forgejo Actions secrets (S3_*, already set) — the uniform CI home (beta's deploy user can't read /etc/thermograph.env) — and mirrored into the SOPS vault for host-side use.
  • deploy/backup/README.md: DR restore runbook (prod DB + Forgejo).

Verify after merge: workflow_dispatch the "Ops cron (backup + IndexNow)" workflow and confirm the backup + forgejo-backup jobs are green and new objects land in the bucket.

Note: beta's deploy user is in the docker group (so docker exec/docker run need no sudo) but has no sudo, so rclone/age are pre-installed there rather than apt-installed by the job.

Delivers the storage-integration plan's backup tier. **Already live** (not just proposed): first backups are in the bucket and restore-verified. **Done manually + verified this session** - Prod DB: 3 dumps age-encrypted to `backups/db/prod/` (~780 MB each). **Restore-verified** (download -> age-decrypt with host key -> valid `PGDMP`). - Forgejo (previously ZERO backups): db + data volume to `backups/forgejo/` (encrypted). - rclone + age installed on prod & beta; Contabo creds validated (path-style, region=default). **This PR makes it recurring** - `ops-cron.yml`: the prod pg_dump job now age-encrypts + pushes off-box; a new `forgejo-backup` job backs up Forgejo's db + data volume on beta. 30-day off-box retention on both. Streamed (`age | rclone rcat`) — no local intermediate, nothing plaintext leaves the box. - Encrypted to the vault age recipient, so `/etc/thermograph/age.key` on each host decrypts for restore. - S3 creds are **Forgejo Actions secrets** (`S3_*`, already set) — the uniform CI home (beta's deploy user can't read `/etc/thermograph.env`) — and mirrored into the SOPS vault for host-side use. - `deploy/backup/README.md`: DR restore runbook (prod DB + Forgejo). **Verify after merge:** `workflow_dispatch` the "Ops cron (backup + IndexNow)" workflow and confirm the `backup` + `forgejo-backup` jobs are green and new objects land in the bucket. **Note:** beta's `deploy` user is in the docker group (so `docker exec`/`docker run` need no sudo) but has no sudo, so `rclone`/`age` are pre-installed there rather than apt-installed by the job.
admin_emi added 1 commit 2026-07-23 20:42:38 +00:00
Add off-box encrypted backups to Contabo Object Storage
All checks were successful
secrets-guard / encrypted (pull_request) Successful in 7s
Details
PR build (required check) / changes (pull_request) Successful in 8s
Details
PR build (required check) / build-backend (pull_request) Has been skipped
Details
PR build (required check) / build-frontend (pull_request) Has been skipped
Details
PR build (required check) / validate-observability (pull_request) Has been skipped
Details
PR build (required check) / gate (pull_request) Successful in 3s
Details
6086b40fe6 
Prod DB dumps and Forgejo (db + data volume) are now copied off-box to the
era5-thermograph S3 bucket, age-encrypted to the vault recipient so the host
age key at /etc/thermograph/age.key decrypts them for restore. Nothing plaintext
leaves the box; no local intermediate (streamed via age | rclone rcat).

ops-cron: the prod pg_dump job gains an off-box push; a new forgejo-backup job
dumps Forgejo's Postgres + tars its data volume on beta (the git host had NO
backup at all). 30-day off-box retention on both. S3 creds are Forgejo Actions
secrets (S3_ENDPOINT/BUCKET/ACCESS_KEY/SECRET_KEY), mirrored into the SOPS vault
(prod.yaml/beta.yaml) for host-side use. Contabo needs path-style addressing.

Adds deploy/backup/README.md with the DR restore runbook.
admin_emi added 1 commit 2026-07-24 20:07:32 +00:00
Merge remote-tracking branch 'origin/dev' into feature/storage-backups-merged
All checks were successful
PR build (required check) / changes (pull_request) Successful in 6s
Details
secrets-guard / encrypted (pull_request) Successful in 7s
Details
PR build (required check) / build-backend (pull_request) Has been skipped
Details
shell-lint / shellcheck (pull_request) Successful in 7s
Details
PR build (required check) / build-frontend (pull_request) Has been skipped
Details
PR build (required check) / validate-observability (pull_request) Has been skipped
Details
PR build (required check) / gate (pull_request) Successful in 2s
Details
e91b49a000 
# Conflicts:
#	infra/deploy/secrets/beta.yaml
#	infra/deploy/secrets/prod.yaml
admin_emi merged commit 519541c378 into dev 2026-07-24 20:08:14 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
  
Compat/Breaking

Breaking change that won't be backward compatible

  
Kind/Bug

Something is not working

  
Kind/Documentation

Documentation changes

  
Kind/Enhancement

Improve existing functionality

  
Kind/Feature

New functionality

  
Kind/Security

This is security issue

  
Kind/Testing

Issue or pull request related to testing

  
Priority
Critical
The priority is critical

  
Priority
High
The priority is high

  
Priority
Low
The priority is low

  
Priority
Medium
The priority is medium

  
Reviewed
Confirmed
Issue has been confirmed

  
Reviewed
Duplicate
This issue or pull request already exists

  
Reviewed
Invalid
Invalid issue

  
Reviewed
Won't Fix
This issue won't be fixed

  
Status
Abandoned
Somebody has started to work on this but abandoned work

  
Status
Blocked
Something is blocking this issue or pull request

  
Status
Need More Info
Feedback is required to reproduce issue or to continue work

No labels
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Jinemi/thermograph#14
No description provided.
Delete branch "feature/storage-backups"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?