alerting: don't print a bare "value:" when ValueString is empty
All checks were successful
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / changes (pull_request) Successful in 11s
shell-lint / shellcheck (pull_request) Successful in 7s
PR build (required check) / build-backend (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Has been skipped
PR build (required check) / validate-observability (pull_request) Successful in 23s
PR build (required check) / gate (pull_request) Successful in 2s

The Discord template printed `value: {{ .ValueString }}` unconditionally, so a
notification carrying no value rendered as "value:" with nothing after it —
which reads as a value that failed to compute rather than one that was never
applicable.

ValueString is only populated when the notification came from an evaluation
that produced refIds. An instance resolved by Grafana's staleness handling has
none: no evaluation returns it again, so the state manager expires it. That is
triggered by a rule title change, since the title becomes the alertname label
and the old label set is orphaned — renaming ProdWorkerContainerSilent to
ProdWorkerHeartbeatMissing did it and posted exactly that message.

Guarding the whole line rather than the substitution: an unguarded
{{ .ValueString }} is harmless on its own, printing the label is the bug.

Verified by executing both templates through text/template with ValueString
set and empty: parses and executes clean, populated output is byte-identical
to the previous format, empty case omits the line.
This commit is contained in:
centralis-agent 2026-07-25 11:15:32 -07:00
parent 1a56a63800
commit 9d5e87b594

View file

@ -37,12 +37,26 @@ contactPoints:
# Deliberately plain. A template error here breaks EVERY notification # Deliberately plain. A template error here breaks EVERY notification
# silently, so this uses only functions verified against Grafana # silently, so this uses only functions verified against Grafana
# 11.6.1 via /api/alertmanager/grafana/config/api/v1/receivers/test. # 11.6.1 via /api/alertmanager/grafana/config/api/v1/receivers/test.
#
# `.ValueString` is guarded because it is not always populated. It
# carries the evaluated refIds only when the notification came from an
# evaluation that produced them; an instance resolved by Grafana's
# STALENESS handling — no evaluation ever returns it again, so the
# state manager expires it — resolves with an empty ValueString. That
# happens whenever a rule's `title` changes, since the title becomes
# the alertname label and the old label set is orphaned. Renaming
# ProdWorkerContainerSilent -> ProdWorkerHeartbeatMissing on
# 2026-07-25 did exactly that and posted a resolved notice reading
# "value:" with nothing after it. An empty labelled field is worse
# than an absent one: it reads as a value that failed to compute.
# Unguarded `{{ .ValueString }}` on its own would be safe; printing
# the label unconditionally is the bug.
message: |- message: |-
{{ range .Alerts }}**severity:** {{ .Labels.severity }} · **host:** {{ .Labels.host }} {{ range .Alerts }}**severity:** {{ .Labels.severity }} · **host:** {{ .Labels.host }}
{{ .Annotations.summary }} {{ .Annotations.summary }}
{{ .Annotations.description }} {{ .Annotations.description }}
`value: {{ .ValueString }}` {{ if .ValueString }}`value: {{ .ValueString }}`
{{ end }} {{ end }}{{ end }}
<https://dashboard.thermograph.org/alerting/list> <https://dashboard.thermograph.org/alerting/list>
# --- The factory-default contact point ------------------------------------------- # --- The factory-default contact point -------------------------------------------