From 9d5e87b594f3d5f13953531775d99086a55bfeb6 Mon Sep 17 00:00:00 2001 From: centralis-agent Date: Sat, 25 Jul 2026 11:15:32 -0700 Subject: [PATCH] alerting: don't print a bare "value:" when ValueString is empty MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Discord template printed `value: {{ .ValueString }}` unconditionally, so a notification carrying no value rendered as "value:" with nothing after it — which reads as a value that failed to compute rather than one that was never applicable. ValueString is only populated when the notification came from an evaluation that produced refIds. An instance resolved by Grafana's staleness handling has none: no evaluation returns it again, so the state manager expires it. That is triggered by a rule title change, since the title becomes the alertname label and the old label set is orphaned — renaming ProdWorkerContainerSilent to ProdWorkerHeartbeatMissing did it and posted exactly that message. Guarding the whole line rather than the substitution: an unguarded {{ .ValueString }} is harmless on its own, printing the label is the bug. Verified by executing both templates through text/template with ValueString set and empty: parses and executes clean, populated output is byte-identical to the previous format, empty case omits the line. --- .../provisioning/alerting/contact-points.yml | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/observability/grafana/provisioning/alerting/contact-points.yml b/observability/grafana/provisioning/alerting/contact-points.yml index 42f1f10..bdddbe2 100644 --- a/observability/grafana/provisioning/alerting/contact-points.yml +++ b/observability/grafana/provisioning/alerting/contact-points.yml @@ -37,12 +37,26 @@ contactPoints: # Deliberately plain. A template error here breaks EVERY notification # silently, so this uses only functions verified against Grafana # 11.6.1 via /api/alertmanager/grafana/config/api/v1/receivers/test. + # + # `.ValueString` is guarded because it is not always populated. It + # carries the evaluated refIds only when the notification came from an + # evaluation that produced them; an instance resolved by Grafana's + # STALENESS handling — no evaluation ever returns it again, so the + # state manager expires it — resolves with an empty ValueString. That + # happens whenever a rule's `title` changes, since the title becomes + # the alertname label and the old label set is orphaned. Renaming + # ProdWorkerContainerSilent -> ProdWorkerHeartbeatMissing on + # 2026-07-25 did exactly that and posted a resolved notice reading + # "value:" with nothing after it. An empty labelled field is worse + # than an absent one: it reads as a value that failed to compute. + # Unguarded `{{ .ValueString }}` on its own would be safe; printing + # the label unconditionally is the bug. message: |- {{ range .Alerts }}**severity:** {{ .Labels.severity }} · **host:** {{ .Labels.host }} {{ .Annotations.summary }} {{ .Annotations.description }} - `value: {{ .ValueString }}` - {{ end }} + {{ if .ValueString }}`value: {{ .ValueString }}` + {{ end }}{{ end }} # --- The factory-default contact point -------------------------------------------