From 24b7ef4c8669865c62b6dcef02f34ba990614516 Mon Sep 17 00:00:00 2001 From: Emi Griffith Date: Sun, 26 Jul 2026 00:16:36 -0700 Subject: [PATCH] runbook: chown the new Caddy log before reloading MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `caddy validate` run as root creates the log files the config names, owned by root. The caddy user then cannot open them and the reload fails with 'permission denied' on the log writer — which is exactly what happened on the live cutover, and what the root-owned centralis.log in that directory is a fossil of. Caddy rejected the config atomically and kept serving prod on the old one, so nothing broke; but the step reads as a no-op until it isn't. --- infra/deploy/RUNBOOK-vps1-vps2-cutover.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/infra/deploy/RUNBOOK-vps1-vps2-cutover.md b/infra/deploy/RUNBOOK-vps1-vps2-cutover.md index 3bbfcd6..fdb379a 100644 --- a/infra/deploy/RUNBOOK-vps1-vps2-cutover.md +++ b/infra/deploy/RUNBOOK-vps1-vps2-cutover.md @@ -247,12 +247,23 @@ serving; nothing has moved yet. ``` ssh agent@169.58.46.181 sudo caddy validate --config /etc/caddy/Caddyfile # BEFORE reloading +# `validate` runs as root and CREATES any log file the config names, owned by +# root — which the caddy user then cannot open, and the reload fails. Fix the +# ownership before reloading, or the first reload rejects the whole config: +sudo chown caddy:caddy /var/log/caddy/beta.log sudo systemctl reload caddy ``` Validate before reload, always: a malformed Caddyfile takes `thermograph.org` down with it, and prod is on this box now. +The `chown` is not optional and it bit this cutover. `caddy validate` as root +left `/var/log/caddy/beta.log` as `root:root`, so the reload failed with +`permission denied` on the log writer. (The root-owned `centralis.log` sitting +in that directory is the same mistake, made earlier.) Note what did NOT happen: +Caddy rejects a bad config atomically and keeps serving the old one, so +`thermograph.org` stayed up throughout — verify it anyway. + Caddy cannot issue the certificate until DNS moves (step 7), so expect the beta hostname to fail TLS until then. That is fine and expected.