Jinemi/thermograph
4
0
Fork 0
Code Issues 2 Pull requests 9 Projects Releases Packages Wiki Activity Actions

UI events v2: consented session tier, failed-search capture, privacy rewrite

Browse source 
Three operator decisions change v1's premises, so the design is now two tiers
with separate flags:

Tier A (THERMOGRAPH_EVENTS) is v1 unchanged -- anonymous hourly aggregates, no
identifier, nothing stored on the device, no consent needed, runs for everyone.
It stays the primary signal precisely so a poor consent rate cannot take the
numbers down with it, and so Tier B's rates can be calibrated against it to
measure the consent bias rather than ignoring it.

Tier B (THERMOGRAPH_EVENT_SESSIONS) adds an ephemeral per-tab id -- 16 random
bytes in sessionStorage, rotating on a 30-minute idle and a 2-hour absolute cap,
capped at 200 events, never linked to another session, device, or to the account
(api_event does not read the auth cookie and there is no user column). Rows land
in a 30-day hypertable with minute-granularity timestamps and an in-session
sequence number instead of precise clock times. Storing an identifier engages
ePrivacy Art. 5(3) and analytics is not strictly necessary, so it is gated on
opt-in consent: an equal-weight banner that mints nothing before "Allow",
one-click withdrawal in every footer that drops the live id immediately, and
GPC/DNT treated as a refusal already given.

THERMOGRAPH_SEARCH_MISS captures zero-result search text server-side in
api_suggest only -- normalised, rejected outright on any personal-data smell,
stored as a per-day count, and pruned below a three-person floor after a week.

The privacy page is rewritten rather than deferred: "no per-visitor identifier"
becomes false the moment Tier B ships. Tests assert the load-bearing promises so
the copy and the code cannot drift apart silently.

Raw-IP truncation is NOT implemented here (the logging pipeline owns it) but
UI-EVENTS.md states what the app side must do, and records that Caddy's default
JSON log already stores full request URIs -- so every search query is in Loki
with the client IP today, which the search-miss mitigations depend on fixing.

UI-EVENTS.md carries the v1-to-v2 diff, the consent reasoning including why
legitimate interest is not available, and an explicit argument that the session
identifier is the wrong trade at this traffic volume.
This commit is contained in:
Emi Griffith 2026-07-23 16:11:06 -07:00
parent 0ddc457d8c
commit 35b58300cb
20 changed files with 1506 additions and 236 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

604
UI-EVENTS.md
View file

@ -1,13 +1,44 @@
# UI product-event instrumentation — design # UI product-event instrumentation — design (v2)
**Status: design + prototype. Feature-flagged OFF. Not deployed anywhere.** **Status: design + prototype. Every tier feature-flagged OFF. Not deployed
Enabling it in any environment requires the operator decisions in anywhere.** Enabling any of it requires the decisions in
[§6](#6-decisions-needed-from-the-operator) first. [§6](#6-decisions-still-needed-from-the-operator) and the privacy-page copy in
[§7](#7-privacy-page-copy) to ship first.
The goal is to learn what people actually *do* in the Thermograph UI — did they The goal is to learn what people actually *do* in the Thermograph UI — did they
ever get a location, does search work, which views earn their maintenance — with ever get a location, does search work, which views earn their maintenance.
instrumentation that is *structurally incapable* of tracking a person, so it
needs no consent banner and contradicts nothing on the public privacy page. **v2 splits the answer into two tiers**, because the operator's decisions
changed what is being built:
* **Tier A — anonymous aggregates.** No identifier, nothing stored on the
device, no consent needed, runs for every visitor. This was the whole of v1
and is still the primary signal.
* **Tier B — consented session rows.** An ephemeral per-tab identifier so
funnels can be computed. Opt-in consent, short retention, its own flag.
Keeping them separate is the single most important structural choice in v2: if
consent rates are poor (they usually are), Tier A still answers most of §1, and
Tier B's numbers can be *calibrated against* Tier A to measure the consent bias
rather than pretending it isn't there.
---
## What changed from v1, and why
| # | v1 | v2 | Why |
|---|---|---|---|
| 1 | **No identifier at all.** Counts of interactions, never of people. | **Ephemeral per-tab session id**, opt-in, 30 min idle / 2 h absolute rotation, sessionStorage. Tier A unchanged and still runs for everyone. | Operator decision 1: rates can't answer "did *this visitor* succeed". |
| 2 | No consent banner needed (nothing stored on the device). | **Opt-in consent banner required** for Tier B, with equal-weight Allow/No, one-click withdrawal in the footer of every page, GPC/DNT treated as a refusal already given. | Storing an identifier engages ePrivacy Art. 5(3); analytics is not "strictly necessary". §5.1 argues the exemption case and rejects it. |
| 3 | `ui_event_hourly` aggregate only — deliberately incapable of holding a behavioural sequence. | Adds `ui_event_session` (one row per event), **30-day retention**, minute-granularity timestamps, `seq` for ordering, 200 events/session cap. | An aggregate cannot express a funnel. This *is* the sequence dataset v1 avoided, so it gets the shortest retention in the system. |
| 4 | Failed search text never captured (§6.2 deferred it). | **Captured server-side** in `api_suggest`: normalised, rejected on any personal-data smell, stored as a per-day count, k-anonymity prune at 3. | Operator decision 2. |
| 5 | Raw IP in `audit.log_access` flagged as an open question. | Treated as **decided: truncate**. Not implemented here (logging-pipeline work owns it); §5.4 states what the app side must do so the two changes stay consistent. | Operator decision 3. |
| 6 | Privacy-page rewrite listed as a follow-up. | **Written and included** (`frontend/templates/privacy.html.j2`), with tests asserting the load-bearing promises. | The v1 page becomes untrue the moment Tier B ships. |
| 7 | One flag, `THERMOGRAPH_EVENTS`. | Three: `THERMOGRAPH_EVENTS` (Tier A), `THERMOGRAPH_EVENT_SESSIONS` (Tier B), `THERMOGRAPH_SEARCH_MISS` (free text). | Each is a separate decision with a separate risk profile and should be separately revocable. |
**Unchanged from v1:** the seven events and their allowlists, the beacon
transport, the abuse-resistance layers, the server-side-derivation preference,
and the rejected storage alternatives.
--- ---
@ -42,6 +73,12 @@ Loki retention is **30 days** (`observability/loki/config.yml`, `720h`), not 24h
and the Alloy glob `/applogs/**/*.jsonl` already picks up the `activity/` stream, and the Alloy glob `/applogs/**/*.jsonl` already picks up the `activity/` stream,
so a new JSONL sink needs no shipping config. so a new JSONL sink needs no shipping config.
A third, found while designing v2 and material to decision 2: the `log` block in
`infra/deploy/Caddyfile` sets no `format`, so Caddy's **default JSON logger**
records `request.uri` — the full URI **including the query string**. Every
`/api/v2/suggest?q=…` a visitor types is therefore already in Loki for 30 days,
next to their IP. See §5.4.
--- ---
## 1. The events worth capturing ## 1. The events worth capturing
@ -64,15 +101,21 @@ someone would actually make:
**Deliberately not captured** — because the request log already answers them, and **Deliberately not captured** — because the request log already answers them, and
duplicating a signal means two numbers that will eventually disagree: page-view duplicating a signal means two numbers that will eventually disagree: page-view
counts, API latency/error rates, and asset traffic. Also deliberately not counts, API latency/error rates, and asset traffic. Also deliberately not
captured: hover/scroll/dwell (needs a session identifier to mean anything), captured: hover/scroll/dwell (individuating, and answers nothing on this list),
map pan/zoom (very high volume, low decision value, and pan/zoom coordinates map pan/zoom (very high volume, low decision value, and pan/zoom coordinates
are location data), and anything at all about *which* place was chosen. are location data), and anything at all about *which* place was chosen.
**v2 note:** the event list is unchanged. Tier B adds a session id to these same
seven events; it does not add an eighth. That is deliberate — the consent
conversation is about *linking* what was already being counted, not about
collecting more, and it keeps the privacy-page copy describing one list.
### Two events are recorded server-side, not in the browser ### Two events are recorded server-side, not in the browser
`place.search` is recorded in `api_suggest`: the server already knows the answer, `place.search` is recorded in `api_suggest`: the server already knows the answer,
every surface that searches is covered for free, and **the query string never has every surface that searches is covered for free, and **the query string never has
to leave the request handler** — only the three-way outcome is stored. Same to leave the request handler** — only the three-way outcome is stored, plus (v2,
decision 2) the normalised text of the *misses* only. Same
argument would apply to alert creation (`accounts/api_accounts.py`). Prefer this argument would apply to alert creation (`accounts/api_accounts.py`). Prefer this
whenever the server can derive the fact: less client code, no forgeability, and whenever the server can derive the fact: less client code, no forgeability, and
strictly less data in flight. strictly less data in flight.
@ -108,9 +151,49 @@ comfort, compare_add, compare_remove, filter_sheet, day_step.
Numbers are bucketed **client-side** before they are sent (`5plus`, not `7`): Numbers are bucketed **client-side** before they are sent (`5plus`, not `7`):
a raw count is a small amount of entropy, and a bucket is all the decision needs. a raw count is a small amount of entropy, and a bucket is all the decision needs.
**Storage row:** `(hour_bucket, event, view, prop, value, referrer) → count`. **Tier A storage row:** `(hour_bucket, event, view, prop, value, referrer) →
There is no row per interaction, and no column an identifier could go in. count`. No row per interaction, no column an identifier could go in. Worst-case
Worst-case key space is a few hundred rows/hour; realistically ~50. key space is a few hundred rows/hour; realistically ~50.
**Tier B storage row:** `(minute_ts, session, seq, event, view, prop, value,
referrer)` — one row per event, written *only* when the visitor consented. The
schema above is unchanged; the session id and sequence number are the only
additions, and no new dimension is collected because of Tier B.
### The Tier B identifier, specified exactly
| | |
|---|---|
| **Value** | 16 bytes from `crypto.getRandomValues`, base64url, 22 chars. Server-validated by exact shape (`^[A-Za-z0-9_-]{22}$`) — see below. |
| **Where it lives** | `sessionStorage`, key `thermograph:sid`. Not `localStorage` (would make it a stable pseudonym across visits — the exact thing being avoided). Not a cookie (would ride on every request including static assets, be readable on routes unrelated to analytics, and buy nothing). |
| **Lifetime** | Rotates after **30 min idle**; rotates after **2 h absolute** regardless of activity; dies with the tab. |
| **Scope** | One tab. Two tabs = two sessions; a return visit = a new session; a second device = unrelated. **No cross-session, cross-tab, or cross-device linkage exists to break** — there is nothing stored that could join them. |
| **Cap** | 200 events per session, then Tier B stops and Tier A carries on. Bounds both storage and how detailed any one sequence can get. |
| **Minted** | Only inside `hasConsent()`. Refusing, ignoring the banner, or signalling GPC/DNT writes *nothing at all* to the device. |
| **Account linkage** | **Never.** Argued below. |
| **Retention** | 30 days, then the chunk is dropped. |
**Why the session id is never joined to the account id** (the decision said
"unless you argue otherwise" — the argument is for *never*):
1. Joining converts an anonymous behavioural sequence into personal data about an
*identified* person, which attaches subject-access, rectification and erasure
duties to the entire event history and makes the 30-day retention a floor
rather than a nicety.
2. The analytical gain is close to zero: every question in §1 is about the
*product* ("does the picker convert?"), not about *who*.
3. Signed-in visitors are a small minority, so an account-linked subset is a
biased sample you could not generalise from anyway — it would be the *most*
engaged users, i.e. exactly the ones whose funnel is least representative.
4. It is enforced structurally, not by policy: `api_event` never reads the auth
cookie, `sendBeacon` cannot attach credentials cross-origin, and there is no
user column in `ui_event_session`. There is a test for it.
**The session field is the only client-controlled string that reaches storage**,
so it is validated by shape and dropped *whole* on any mismatch rather than
sanitised into shape. A trimmed-to-fit id would still be attacker-chosen text; a
rejected one is nothing. Tier A records identically either way, so refusing a
malformed id costs the primary signal zero.
Migration note: the four live `home.*` events keep their current unflagged, Migration note: the four live `home.*` events keep their current unflagged,
un-dimensioned behaviour so the ops endpoint's numbers don't reset when this un-dimensioned behaviour so the ops endpoint's numbers don't reset when this
@ -135,6 +218,17 @@ as a follow-up, once there's overlap data to stitch the series.
* **Live sink** — one `audit.log_activity("ui.event", …)` JSONL line per event. * **Live sink** — one `audit.log_activity("ui.event", …)` JSONL line per event.
Alloy already ships that path, so the last 30 days are queryable in Grafana Alloy already ships that path, so the last 30 days are queryable in Grafana
immediately without waiting on the rollup. Same allowlisted enums, nothing else. immediately without waiting on the rollup. Same allowlisted enums, nothing else.
* **Tier B sink** — `ui_event_session`, one row per event, 1-day chunks, **30-day**
retention policy. Timestamps truncated to the **minute** and ordering carried by
an in-session `seq` instead: precise clock times are a fingerprint (millisecond
inter-event gaps individuate, and adjacency lets two rotated sessions be
stitched back together), and a funnel needs order, not timing. No IP column, no
user-agent column, no account column, and no unique constraint that would
require one.
* **Rollup** — at 30 days, per-event rows are aggregated into anonymous
session-outcome rows (`day, entry_view, reached_pick, reached_grade, n`) with no
identifier, retained alongside Tier A. The sequence data expires; the learning
does not. *(Specified here, not yet implemented — see §7.)*
* **The existing in-process counters stay untouched** — they remain the live * **The existing in-process counters stay untouched** — they remain the live
"what's happening right now" view on `/api/v2/metrics`. "what's happening right now" view on `/api/v2/metrics`.
@ -147,11 +241,20 @@ as a follow-up, once there's overlap data to stitch the series.
* *Extend the in-process counters.* Cheapest option and no new storage, but they * *Extend the in-process counters.* Cheapest option and no new storage, but they
are UNLOGGED, TRUNCATEd every deploy, capped at 7 days, and have exactly one are UNLOGGED, TRUNCATEd every deploy, capped at 7 days, and have exactly one
dimension. Every question in §1 needs at least two. Kept, not extended. dimension. Every question in §1 needs at least two. Kept, not extended.
* *Per-event rows in a hypertable (a real firehose).* With no identifier, a row * *Per-event rows for **everyone** (a real firehose).* Still rejected. v2 adds
per interaction gives you nothing an hourly aggregate doesn't — except per-event rows, but only for the consented minority and only for 30 days.
intra-hour timing, which no listed decision needs. It costs a table that Making them the default would put every visitor's behavioural sequence in the
*could* hold a behavioural sequence, which is exactly the property that would database to answer questions that Tier A's aggregate already answers, and
make the privacy analysis hard. Rejected on privacy grounds, not cost. would drag the whole dataset behind the consent banner.
* *Session-aggregate instead of per-event rows for Tier B* (one row per session:
`entry_view, reached_pick, reached_grade, n_controls`). Genuinely tempting —
it is strictly less data and answers the headline funnel. Rejected for the
**live** table because the client would have to decide when a session is
"done" (it never reliably knows — tabs are closed, not ended) and because a
fixed set of outcome columns forecloses any funnel question not thought of in
advance. It *is* adopted as the **rollup**: at 30 days the per-event rows are
aggregated into exactly this shape, with no session id, and kept 400 days —
so the sequence data is short-lived and the learning is permanent.
* *A third-party analytics SaaS (Plausible/Umami/GA).* GA is a non-starter * *A third-party analytics SaaS (Plausible/Umami/GA).* GA is a non-starter
(US transfer, consent banner, cookies). Self-hosted Plausible/Umami would be (US transfer, consent banner, cookies). Self-hosted Plausible/Umami would be
defensible, but it means another service to run and its own identifier model defensible, but it means another service to run and its own identifier model
@ -191,6 +294,19 @@ crawlers and scanners. Layered, in order of how much they actually buy:
false confidence. The structural defence is that the beacon requires JS false confidence. The structural defence is that the beacon requires JS
execution and is linked from nowhere. execution and is linked from nowhere.
7. **Session-field shape validation** (new in v2). `session` is the one
client-controlled string that reaches storage. It is matched against
`^[A-Za-z0-9_-]{22}$` and dropped whole on mismatch — never trimmed into
shape, because trimmed attacker text is still attacker text. A forged id also
cannot inflate anything: Tier A is keyed on enums, not on sessions.
8. **Unbounded session minting** is the one new abuse surface — an attacker can
generate as many valid-looking ids as they like and manufacture fake
"sessions". The per-IP rate limit bounds it to 120 rows/min/IP, the 200-event
cap bounds any single session, and the 30-day retention bounds the blast
radius. The detection signal is the same cross-check as below: Tier B's
session count should stay in a stable ratio to Tier A's `view.open` count, and
a sudden divergence means manufactured sessions.
**Poisoning** (someone inflating a number rather than exhausting storage) cannot **Poisoning** (someone inflating a number rather than exhausting storage) cannot
be fully prevented without identity. Mitigation is cross-checking: the server's be fully prevented without identity. Mitigation is cross-checking: the server's
own inbound page counts are an independent measure of `view.open`, so a own inbound page counts are an independent measure of `view.open`, so a
@ -202,179 +318,355 @@ divergence between them is the alarm. Any analysis should treat these numbers as
## 5. Privacy position ## 5. Privacy position
The site is EU-hosted with EU visitors, so GDPR + ePrivacy apply. The public The site is EU-hosted with EU visitors, so GDPR + ePrivacy apply. The public
privacy page currently promises, verbatim: *"no analytics library, no cookies for privacy page promised, verbatim: *"no analytics library, no cookies for tracking,
tracking, and no per-visitor identifier… simple aggregate counts… with nothing and no per-visitor identifier… simple aggregate counts."* **Tier B makes that
tying any of it to an individual."* That is a commitment already made, and this false**, which is why the rewritten copy (§7) is part of this change and not a
design is built to stay inside it rather than to renegotiate it. follow-up.
**What the design does:** ### 5.1 Consent — the argument, and why I land on "banner"
* **No identifier of any kind.** No cookie, no `localStorage`/`sessionStorage` The question is whether Tier B can run without opt-in. It cannot, and here is the
id, no fingerprint, no hashed IP+UA. Not even the logged-in user id: the reasoning rather than an assumption.
handler never reads the session cookie, and `sendBeacon` cannot attach
credentials cross-origin anyway.
* **Nothing is written to the visitor's device**, so ePrivacy Art. 5(3)
(the "cookie rule", which covers *any* storage or access on terminal equipment,
not just cookies) is not engaged at all.
* **No IP is stored** by this feature — used as a per-minute rate-limit key and
discarded.
* **No free text, no coordinates, no URLs.** `place.pick` records *how* a
location was chosen, never where. The referrer is reduced server-side to a bare
registrable domain, never a full URL (which can carry someone's search query).
* **Aggregate-only storage.** An hour bucket and a tuple of enums; no row per
interaction, so no behavioural sequence exists to be reconstructed.
* **GPC and DNT are honoured** — a visitor signalling either sends nothing.
**Consequence to be honest about:** with no identifier there are **no unique **ePrivacy Art. 5(3) is the binding constraint, not GDPR.** It covers *any*
visitors, no sessions, no funnels, no bounce rate, no returning-visitor rate**. storing of, or access to, information on terminal equipment — not just cookies,
`view.open` counts view opens, not people. Ratios across events so `sessionStorage` is squarely in scope. It admits exactly two exemptions:
(`place.pick / view.open`) are *rates*, not *conversion*, because the numerator transmission-necessary, and strictly necessary for a service *explicitly
and denominator can't be tied to the same visit. That is the price of not needing requested by the user*. Nobody visits a weather site to be measured, so the
consent, and it is worth it — but it must not be quietly forgotten when someone second exemption does not fit on its plain reading. Crucially, **legitimate
later asks "how many users do we have?" interest is not available here at all** — it is a GDPR Art. 6 lawful basis, and
it cannot substitute for the Art. 5(3) consent requirement. Any argument of the
form "we have a legitimate interest in analytics, so no banner" is simply
answering the wrong question.
**Legal reading (for the operator to confirm, not for me to decide):** with no **The one genuinely arguable route** is the audience-measurement exemption:
identifier, no device storage, no cross-site data and aggregate-only retention, CNIL's guidance (and similar positions from a few other DPAs) exempts a
this is anonymous audience measurement — it fits the CNIL exemption criteria and first-party measurement identifier where it is used *solely* for aggregate
the EDPB's "strictly necessary"/first-party-analytics reasoning, so **no consent statistics, is not shared, is not cross-site, is not combined with other
banner should be required**. Transient IP processing for rate limiting rests on processing, and has bounded lifetimes. This design would fit that description
legitimate interest (Art. 6(1)(f)) and is security-necessary. No DPIA is comfortably — arguably better than the tools the exemption was written for, since
triggered (no systematic monitoring, no profiling, no special categories). One the id here dies in 2 hours rather than 13 months. So the argument is real, and
line should be added to the Art. 30 record of processing. I am not dismissing it.
**A pre-existing issue this surfaced, unrelated to the new events.** **Why I still recommend the banner:**
`audit.log_access` writes the **raw client IP** for every non-static request into
`logs/access/*.jsonl`, which Alloy ships to Loki with 30-day retention. That is 1. It is a **national regulator's exemption, not harmonised EU law**. Germany's
personal data under GDPR, it is retained for 30 days fleet-wide, and it sits TDDDG §25 is read more narrowly, and the EDPB has never blessed a general
awkwardly beside the privacy page's *"not logged beyond the normal web-server analytics exemption. EU visitors are not all French.
request handling"*. **This design does not touch it and does not extend it** — 2. The exemption is conditioned on use — "solely for aggregate statistics". Tier
but it should be decided on (§6.3) rather than left implicit, and it would be B exists precisely to look at *sequences* within a session. That is still
odd to ship privacy-first analytics while the access log keeps raw IPs for a aggregate at the reporting layer, but it is a harder story to tell than a
month. pageview counter, and the conditions are assessed on what you actually do.
3. The downside is asymmetric. A banner costs a UX annoyance and some sample
bias. Getting the exemption wrong on a site with no legal budget costs
correspondence with a regulator.
4. **Tier A makes the banner cheap.** Because the anonymous tier runs regardless,
a refusal costs the *funnels*, not the *numbers*. If Tier A were also behind
the banner I would push much harder on the exemption.
**The consent mechanism** (`frontend/static/consent.js`), designed to be a real
choice rather than a compliance prop:
* Nothing is stored and **no id is minted before an explicit "Allow"** — ignoring
the banner leaves exactly as little behind as refusing it. There is no
"pending" state that quietly collects.
* "Allow" and "No thanks" are the **same size, weight and prominence**. No
pre-ticked boxes, no second-layer "legitimate interest" tab, no cookie-wall.
* It is a **bar, not a modal**: it never blocks reading the page, which is the
product.
* **Withdrawal is as easy as granting** — an `Analytics: on/off` control in the
footer of every page and on the privacy page. Withdrawing deletes the live
session id immediately, discards anything queued-but-unsent, and stops further
recording within the same page load.
* **Re-asking:** a grant is re-confirmed after ~6 months; a refusal is **not**
re-asked for 12 months. A banner that reappears after "no" is a dark pattern,
not a question.
* **GPC / DNT are treated as a refusal already given**: no banner is shown, no id
is minted, and Tier A also stands down. Stricter than the law requires (GPC has
no binding EU status), and it means the most privacy-conscious visitors are
never nagged.
* The **consent record itself** lives in `localStorage`. Storing a user's own
choice is strictly necessary to honour it, so it is exempt from the consent it
records — and it is deliberately the only thing that persists across sessions.
### 5.2 What is collected, tier by tier
| | Tier A (anonymous) | Tier B (consented) |
|---|---|---|
| Consent | not required | **opt-in required** |
| Identifier | none | ephemeral, per-tab, 30 min idle / 2 h absolute |
| On-device storage | none | one `sessionStorage` key |
| Stored per interaction | nothing (hourly counts) | one row |
| IP | never | never |
| Account linkage | never | never |
| Free text | never | never |
| Coordinates | never | never |
| Retention | 400 days | **30 days**, then anonymised rollup |
| Runs for | everyone | consenting visitors only |
### 5.3 Consequences to be honest about
* **The consented sample is self-selected, and the bias is not random.** People
who decline analytics skew technical and privacy-conscious — on a site about
climate percentiles, that is plausibly *the same population* as the power users
whose funnels are most interesting. Tier B may systematically under-represent
exactly the behaviour it was added to study.
* **Mitigation:** every Tier B rate has a Tier A counterpart computed over
everyone. Comparing them measures the consent bias directly. If
`place.pick / view.open` differs materially between the two, Tier B's funnels
should be read as a *lower bound on a subpopulation*, not as the truth.
* **We cannot honour an erasure request for past events**, because the whole
point is that nothing links a session to a person — there is no lookup key.
The mitigation is the 30-day retention, and the privacy page says so plainly
rather than implying a deletion path that does not exist. *(A "forget this
session" beacon carrying the live id would make it possible for the current
session only — specified in §6.6, not built.)*
* **Two tabs are two sessions**, so funnels **undercount** completion. That is
the error direction to prefer, but it must not be forgotten when reading a
number.
### 5.4 IP addresses (decision 3 — not implemented here)
The logging-pipeline work owns this. Recorded here so the two changes stay
consistent:
* **The app side must stop writing raw IPs.** `audit.log_access` currently writes
`_client_ip(request)` verbatim into `logs/access/*.jsonl`, which Alloy ships to
Loki with 30-day retention. It should write a truncated form — **/24 for IPv4,
/48 for IPv6** — or drop the field. Truncation preserves the geo-scale and
abuse-pattern uses while stopping single-device identification.
* **The rate limiter must keep the full IP in memory.** `metrics.rate_ok` needs
per-address granularity; truncating its key to /24 would let one abuser
rate-limit an entire NAT'd office. Full precision in a map cleared every
minute, never written to disk, is the right split.
* **Nothing in this design assumes durable raw IPs.** No event carries one, and
no analysis depends on one. The only cost of truncation is weaker retrospective
forensics if the beacon is ever abused — accepted.
* **Caddy is the bigger leak, and it is in scope for decision 2.** The `log`
block in `infra/deploy/Caddyfile` sets no `format`, so it uses Caddy's default
JSON logger — which records `request.uri`, i.e. **the full URI including the
query string**. Every `/api/v2/suggest?q=<what someone typed>` is therefore
already in Loki for 30 days, alongside the client IP. Capturing failed searches
carefully in the application while Caddy logs *every* search verbatim would be
theatre. The Caddy log needs the query string stripped (or the field removed)
in the same pass.
### 5.5 Legal housekeeping
* **Lawful basis:** consent (Art. 6(1)(a)) for Tier B; Tier A processes no
personal data at the sink; transient IP handling for rate limiting rests on
legitimate interest (Art. 6(1)(f)) and is security-necessary.
* **DPIA:** still not triggered — no systematic large-scale monitoring, no
profiling with legal effect, no special categories, and the scale is ~3k human
requests/day. Worth a one-page note recording *why* it was not triggered.
* **Art. 30 record of processing:** needs two lines now, not one — the anonymous
counts and the consented session statistics, with their separate retentions.
* **Processors:** none. Everything stays on hardware the operator controls.
--- ---
## 6. Decisions needed from the operator ## 6. Decisions still needed from the operator
Nothing below is decided in the prototype. The flag stays off until these are Decisions 1–3 are settled and built (or, for IPs, documented). What follows is
answered. what is *still* open. The flags stay off until these are answered.
**6.1 — Identifier: none, ephemeral session, or per-user?** **6.1 — Approve the privacy-page copy (§7).** It is written and tested but it is
Recommended: **none** (what the prototype implements). a public legal statement in the operator's name. Nothing turns on until it is
*Consequence of "none":* no unique visitors, no funnels, no retention cohorts — approved and shipped. Specifically confirm the numbers in it are the ones you
counts of interactions, not of people. intend to honour: 400 days (anonymous), 30 days (session), 90 days (failed
*Ephemeral session id* (random, `sessionStorage`, 30 min): enables funnels and searches), 30 min / 2 h (id rotation).
sessions. But it is storage on the visitor's device → **ePrivacy consent banner
required**, and it makes the data pseudonymous rather than anonymous
(subject-access/erasure duties attach). Also requires rewriting the privacy page.
*Per-user when logged in:* directly links behaviour to an identified person.
Highest analytical value, highest duty, and hardest to justify for a free weather
site. Recommended answer: **never**, regardless of what is chosen for anonymous
visitors.
**6.2 — Failed search queries: capture the text or not?** **6.2 — Time-box Tier B, or run it permanently?** Strong recommendation:
The single most useful piece of product data here is *what people searched for **time-box it.** Turn it on for a defined study window (6–8 weeks), answer the
when we found nothing*. Three options: (a) never capture — prototype default; specific funnel questions, roll up, then turn the flag off and keep Tier A
(b) capture **zero-result queries only**, server-side, normalised (lowercased, running. The banner then disappears again. A permanent banner is a permanent tax
length-capped, rejected if it contains `@`, a digit run, or a URL-ish token), on every first visit, for a question that is mostly asked once.
30-day retention, never joined to anything; (c) capture all queries — not
recommended, since successful queries are already answered by the `hit` counter
and a search box is a free-text field a person can type anything into.
Recommended: **(b), as a separately flagged follow-up**, with its own privacy-page
sentence. Do not bundle it with the initial rollout.
**6.3 — Raw IPs in the access log (pre-existing).** **6.3 — Failed-search retention and the k-floor.** Built with
Keep as-is / truncate to /24 (IPv4) and /48 (IPv6) / hash with a daily-rotating `SEARCH_MISS_MIN_COUNT = 3` (strings fewer than 3 people typed are deleted after
salt / drop entirely. Recommended: **truncate**, which preserves the geo-scale and 7 days) and 90-day overall retention. Confirm, or tighten. Raising k to 5 costs
abuse-pattern uses while stopping single-device identification. This is little and is the cheapest way to buy margin on the only free-text field.
independent of the new events but should be decided in the same pass.
**6.4 — Consent banner: rely on the analytics exemption, or add one?** **6.4 — Who may read Tier B?** Per-event session rows are the most sensitive
Recommended: **rely on the exemption** — it is only available while 6.1 stays thing the project stores. Today anyone with database access can read them.
"none" and 6.2 stays "(a) or (b)". Adding a banner unlocks identifiers but costs Decide whether that is acceptable or whether a restricted role / view-only
a real chunk of the (small) traffic to banner fatigue and adds a consent-state rollup access is wanted.
machine to every page.
**6.5 — Retention for `ui_event_hourly`.** **6.5 — Environment separation.** Dev and beta run the same code. Recommended:
Set in the migration (`RETENTION_DAYS`, currently **400** — a full year-over-year Tier A on dev during validation then off; Tier B **never** on dev or beta (real
comparison). Anonymous aggregates make this a storage/usefulness question rather consent from real visitors is not obtainable on a staging box, and dev traffic
than a GDPR one, but it should be a deliberate number. Options: 90 / 400 / would pollute the funnels). Confirm.
indefinite.
**6.6 — Privacy-page copy.** **6.6 — "Forget this session" endpoint?** A one-click withdrawal could
The page must keep matching reality *before* the flag goes on: the Analytics additionally send the live session id to a delete endpoint, so the current
section needs a sentence naming what is counted and stating the retention. Who session's rows are erased rather than merely expiring in 30 days. Costs: a new
writes and approves that copy? public mutating endpoint that takes an id (unguessable at 128 bits, but still an
endpoint), and a *stronger* deletion promise than can be kept for older sessions.
Recommended: **yes, as a follow-up**, because "turn it off" that also erases what
was already collected is a materially better promise. Not built.
**6.7 — Environment separation.** **6.7 — Consent re-ask intervals.** Built as ~6 months for a grant, 12 months
Dev and beta run the same code. Should events be recorded there at all, and if before ever re-asking a refusal. Confirm or change.
so, tagged with the environment so they never mix into prod's numbers? (A
`host`/`env` label already exists in Loki; the hypertable has no such column.)
Recommended: **flag on for dev only during validation, then off; prod-only after.**
**6.8 — Referrer beyond `view.open`.** **6.8 — Referrer beyond `view.open`.** Unchanged from v1: kept only on
The prototype keeps a referrer domain only on `view.open`. Widening it to every `view.open`. Widening it would allow "which referral source converts" per session
event would allow "which referral source uses Compare" at the cost of a much — genuinely useful now that Tier B exists, but it also makes a session more
larger key space. Recommended: **leave as-is** until a question needs it. linkable to its arrival context. Recommended: **leave as-is**, and answer it by
joining Tier A day-level referrer counts instead.
**6.9 — GPC/DNT.** **6.9 — Confirm GPC/DNT standing down Tier A as well.** Built that way. It is
The prototype honours both, biasing counts slightly downward. Confirm that trade, stricter than required and costs a little coverage on the tier that needs no
or drop it. consent at all. Confirm, or restrict the GPC/DNT stand-down to Tier B only.
--- ---
## 7. Implementation plan ## 6b. Where I think these decisions are wrong
Asked for judgement, not compliance. Decisions 2 and 3 I agree with. Decision 1
I would push back on.
**Decision 1 (session identifier) — I think this is a bad trade for this
product, at this scale, right now.** Four reasons, in order of weight:
1. **The volume probably cannot support the analysis.** ~3,000 human requests/day
is on the order of 600–900 sessions/day. A plain, non-manipulative banner
typically converts somewhere in the 20–60% band; take 35% and you have
~250 consented sessions/day. A four-step funnel on 250 sessions/day gives step
rates with confidence intervals of several percentage points — so detecting
anything short of a large effect takes weeks per comparison, and any A/B
reading is out of reach entirely. The identifier buys a *shape* you could
mostly have guessed, not a measurement you can act on.
2. **The bias is correlated with the thing being measured** (§5.3). Privacy-
declining users and power users are plausibly the same people here.
3. **The most valuable funnel needs no identifier at all.** "Did the visitor who
tapped *Use my location* reach a grade?" happens entirely within one page
load on the homepage — a single JS context, no navigation, no storage. It is
answerable today with an ordered pair of Tier A events and zero consent. Same
for the picker→grade path. What genuinely needs the id is only the
*cross-view* journey (home → calendar → compare), which is the less pressing
question.
4. **It costs the product's clearest differentiator.** The footer says "No ads.
No tracking." and the privacy page was, unusually, entirely true. Replacing
that with a consent banner is a real product cost on a site whose appeal is
partly that it does not behave like everything else.
**What I would do instead:** ship Tier A, run it for a month, and write down the
specific questions it failed to answer. If any survive, turn Tier B on as a
time-boxed study (§6.2) rather than as permanent instrumentation. The prototype
is built so this costs nothing — Tier B is a separate flag and a separate table,
and turning it on later requires no migration of Tier A data.
**Decision 2 (failed-search text) — agreed, with one correction.** The design
mitigations (server-side only, reject-don't-sanitise, per-day counts, k≥3 prune)
are, I think, sufficient. But **the mitigations are worthless until the Caddy log
is fixed** (§5.4): Caddy currently logs the full request URI including
`?q=<query>` for *every* search, hit or miss, unnormalised, next to the client
IP, for 30 days. The careful in-app capture is strictly less exposure than what
is already happening by accident. Fix the Caddy log format in the same pass or
decision 2 is a net *decrease* in privacy only on paper.
**Decision 3 (drop/truncate IPs) — agreed, no reservations.** The only cost is
weaker retrospective abuse forensics, which is a fair price. Note the rate
limiter must keep full precision in memory (§5.4).
---
## 7. Privacy-page copy
Written and shipped in this change: `frontend/templates/privacy.html.j2`. The
substantive edits from the live page:
* **"Analytics" → "Usage statistics"**, split into *Anonymous counts — always on*
and *Session statistics — only if you say yes*, so the consent boundary is the
organising structure of the section rather than a footnote.
* The claim **"no per-visitor identifier"** is removed — it becomes false.
* The session id is described concretely: session storage not a cookie, current
tab only, 30 min idle / 2 h absolute rotation, gone when the tab closes, never
linked to the account or to an IP, 30-day retention.
* **The withdrawal path is stated**, and so is its limit: turning it off deletes
the id immediately and stops recording, but past records cannot be found and
deleted *because* they are not linked to anyone — with the short retention
given as the reason that is acceptable.
* GPC/DNT are named explicitly as being honoured.
* A new **"Searches that found nothing"** subsection: what is kept, what is
discarded, the k-anonymity prune, and that successful searches are never stored
as text.
* A new **"Server logs"** subsection stating that addresses are shortened before
being written — this must not ship before the logging-pipeline change actually
truncates them, or the page becomes untrue in the other direction.
* The location section now states positively that **coordinates never enter the
usage statistics** — only *how* a place was picked, never where.
* The footer's blanket **"No tracking."** becomes **"No third-party trackers."**
on both the base template and the homepage hero, which is the claim that stays
true.
Tests assert the load-bearing promises (`frontend/tests/unit/test_pages.py`), so
if the copy and the code drift apart the suite fails rather than the page quietly
becoming a false statement.
---
## 8. Implementation plan
### Backend (`backend/`) ### Backend (`backend/`)
| File | Change | Status in prototype | | File | Change | Status |
|---|---|---| |---|---|---|
| `core/events.py` | **new** — `SCHEMA`, `normalize()`, hourly upsert, JSONL sink, `ENABLED` flag | done | | `core/events.py` | `SCHEMA`, `normalize()`, hourly upsert, JSONL sink; **v2:** three flags, `validate_session`/`validate_seq`, session-row insert, `normalize_search_miss` + `record_search_miss`, `PRUNE_SEARCH_MISS_SQL` | done |
| `core/metrics.py` | expose `rate_ok()` so one event spends one token across both sinks; `record_event(rate_limit=False)`; raise the per-IP ceiling to 120/min and make it env-tunable; `reset_rate_limiter()` for tests | done | | `core/metrics.py` | `rate_ok()` so one event spends one token across sinks; per-IP ceiling 120/min, env-tunable; `reset_rate_limiter()` for tests | done |
| `web/app.py` | `api_event`: batched payload, streaming body cap, `Sec-Fetch-Site` check, fan-out to both tiers, still always 204. `api_suggest`: record `place.search` server-side | done | | `web/app.py` | `api_event`: batching, streaming body cap, `Sec-Fetch-Site` check, **v2:** `session`/`seq` pass-through. `api_suggest`: `place.search` + **v2:** `record_search_miss` on zero results | done |
| `alembic/versions/0003_ui_events.py` | **new** — `ui_event_hourly` hypertable + retention policy, Postgres-only (no-ops on the SQLite test bind, like `0002`) | done | | `alembic/…/0003_ui_events.py` | Tier A `ui_event_hourly` hypertable, 400-day retention | done |
| `accounts/api_accounts.py` | record `alert` steps server-side on subscription create/delete | **todo** | | `alembic/…/0004_ui_event_sessions.py` | **v2:** `ui_event_session` hypertable (30-day retention, 1-day chunks) + `ui_search_miss` | done |
| `web/app.py` / `frontend/content.py` | record `deadend prop=not_found` on 404 responses | **todo** | | daily maintenance job | run `events.PRUNE_SEARCH_MISS_SQL`; drop `ui_search_miss` rows past 90 days | **todo** |
| `tests/core/test_events.py` | **new** — allowlist, slot dropping, no-free-text-by-construction, flag-off inertness, never-raises | done (10 tests) | | 30-day rollup job | `ui_event_session` → anonymous session-outcome rows before the retention drop | **todo** |
| `tests/web/test_event_beacon.py` | **new** — uniform 204, batch cap, oversize drop, cross-site drop, coordinates/free text never survive, legacy shape still counts | done (11 tests) | | `accounts/api_accounts.py` | `alert` funnel steps recorded server-side | **todo** |
| `web/app.py` / `content.py` | `deadend prop=not_found` on 404s | **todo** |
| `tests/core/test_events.py` | allowlist, slot dropping, no-free-text-by-construction, flag-off inertness, never-raises; **v2:** session-shape validation, seq bounds, tier independence, id-never-in-logs, search-miss normalisation + rejection + k-floor | done (21) |
| `tests/web/test_event_beacon.py` | 204 uniformity, batch cap, oversize drop, cross-site drop; **v2:** session/seq over the wire, forged-session rejection, sessions-flag independence, auth-cookie never read | done (15) |
| `tests/web/test_api.py` | **v2:** miss text captured server-side only, hits never stored, never in the app access log | done (2) |
### Frontend (`frontend/`) ### Frontend (`frontend/`)
| File | Change | Status | | File | Change | Status |
|---|---|---| |---|---|---|
| `static/track.js` | **new** — flag check, GPC/DNT opt-out, batching + flush on `pagehide`/`visibilitychange`, `currentView()`, `[data-event]` delegation, `trackLegacy()` for the four live events | done | | `static/consent.js` | **v2, new** — opt-in banner, equal-weight buttons, localStorage consent record with re-ask intervals, GPC/DNT stand-down, footer withdrawal control, `onConsentChange` | done |
| `static/digest.js` | beacon moved out; re-exports `track`/`trackLegacy` so importers keep working | done | | `static/track.js` | flag check, batching, `currentView()`, `[data-event]` delegation, `trackLegacy()`; **v2:** consent gate, session minting/rotation/caps, `seq`, immediate drop on withdrawal | done |
| `static/app.js` | `view.open`, `place.pick` (geolocate + picker), `deadend` (geo denied, grade error, offline), `share` (link + png), `view.control chart_metric` | done | | `static/style.css` | **v2:** `.consent-bar` / `.consent-control`, tokens only, 44px targets, mobile stacking | done |
| `static/mappicker.js` | thread a `method` ("search" vs "map") through `finish()` → `onPick(lat, lon, method)` | done | | `static/digest.js` | beacon moved out; re-exports for importers | done |
| `static/units.js` | declarative `data-event` attributes on the °F/°C toggle — no import needed | done | | `static/app.js`, `mappicker.js`, `units.js` | `view.open`, `place.pick`, `deadend`, `share`, `view.control` call sites | done |
| `static/*.html` (6 SPA shells) | load `track.js` | done | | `static/*.html` (6 shells) | load `consent.js` then `track.js` | done |
| `templates/base.html.j2` | stamp `data-tg-events` on `<html>` when enabled | done | | `templates/base.html.j2` | flag stamp on `<html>`; **v2:** consent script, footer withdrawal control, "No third-party trackers" | done |
| `content.py`, `app.py` | `EVENTS_ENABLED`, applied to both HTML paths (Jinja + memoized static shells) | done | | `templates/home.html.j2` | **v2:** hero trust line no longer claims "No tracking" | done |
| `static/calendar.js`, `compare.js`, `day.js`, `score.js`, `subscriptions.js` | `view.open` + their own `view.control` / `place.pick` / `alert` call sites | **todo** | | `templates/privacy.html.j2` | **v2:** full rewrite (§7) | done |
| `templates/privacy.html.j2` | Analytics-section copy update — **blocked on 6.6** | **todo** | | `content.py`, `app.py` | `EVENTS_ENABLED` on both HTML paths | done |
| `tests/unit/test_pages.py` | flag absent by default on every HTML path; stamped on both paths when enabled | done (2 tests) | | `static/calendar.js`, `compare.js`, `day.js`, `score.js`, `subscriptions.js` | their own call sites | **todo** |
| `tests/unit/test_pages.py` | flag off by default / stamped when on; **v2:** consent control present everywhere, script order, privacy-copy promises, footer claim | done (6) |
### Rollout ### Rollout
1. Land with `THERMOGRAPH_EVENTS` **unset everywhere**. The feature is inert: 1. Land with **all three flags unset everywhere**. Inert: the beacon answers 204,
the beacon still answers 204, nothing reaches a sink, and the flag stamp is no sink is written, no banner renders, no id is minted, and the flag stamp is
absent from every page (asserted by test). absent from every page (asserted by test).
2. Resolve §6. Update the privacy page. Nothing turns on before this. 2. **Ship the privacy-page copy and the IP truncation first** — including the
3. Run the migration on dev; enable on **dev only**; verify in Grafana Caddy log-format fix (§5.4). The page must not claim addresses are shortened
(`tag="ui.event"`) that the shape is right and that no free text or coordinate before they are.
ever appears. 3. Migrate + enable **Tier A only** on dev, then beta, then prod. Verify in
4. Enable on beta for a week. Compare `view.open` against the middleware's own Grafana (`tag="ui.event"`) that no free text or coordinate ever appears.
page counts — a large divergence means bot noise or a client bug, and is the 4. Run Tier A for a month. Write down which questions it failed to answer (§6b).
go/no-go for prod. 5. Only then, and only if step 4 produced surviving questions: enable
5. Enable on prod. Re-check the `other` bucket and the `Sec-Fetch-Site`-less `THERMOGRAPH_SEARCH_MISS` and `THERMOGRAPH_EVENT_SESSIONS` on **prod only**,
bucket weekly for the first month; a growing `other` means either a schema gap as a time-boxed study. Watch the consent accept rate and compare Tier B's
or someone poking the endpoint. rates against Tier A's to quantify the bias before trusting any funnel.
6. Kill switch: unset the flag. No deploy, no migration, no data loss — existing 6. Roll up and turn Tier B off at the end of the window; the banner disappears
rows just stop growing. with it.
7. Kill switch: unset a flag. No deploy, no migration, no data loss — and
unsetting `THERMOGRAPH_EVENT_SESSIONS` alone removes the banner and the
identifier while leaving the anonymous history intact.
### Cost check ### Cost check
~3,000 human req/day → on the order of 1–3k events/day, batched ≈ a few hundred ~3,000 human req/day → ~1–3k events/day, batched ≈ a few hundred extra
extra requests/day (~0.005 req/s) and a handful of upserts. Negligible against a requests/day (~0.005 req/s). Tier A grows a few hundred rows/day. Tier B, at a
48 GB box; the hypertable grows by roughly a few hundred rows/day, well under 35% consent rate and ~10 events/session, is on the order of 2–3k rows/day held
100 MB/year before compression. for 30 days — under a million rows at steady state, trivially small. `ui_search_miss`
is bounded by distinct failed queries per day, realistically tens.

98
backend/alembic/versions/0004_ui_event_sessions.py Normal file
View file

@ -0,0 +1,98 @@
"""UI events tier B: consented session rows + failed-search text
Two tables, both created together because they arrived from the same pair of
operator decisions (see UI-EVENTS.md §6):
* ``ui_event_session`` — one row per event, carrying an ephemeral per-tab session
id, so a funnel can be computed. This is a **behavioural sequence dataset**:
the thing ``ui_event_hourly`` was deliberately shaped to avoid. It therefore
gets the shortest retention in the system (30 days), minute-granularity
timestamps rather than precise ones, and an in-session ``seq`` to carry order.
Only ever written for a visitor who gave opt-in consent.
* ``ui_search_miss`` — normalised zero-result search text, the only free-text
field anywhere in the instrumentation, captured server-side only. Stored as a
per-day count rather than a row per search, and pruned below a k-anonymity
floor (see ``core/events.py`` SEARCH_MISS_MIN_COUNT) so a string one person
typed once does not persist.
Retention numbers live here, not in a hand-typed policy on the box, so they are
reviewable in the repo and move with the code.
Postgres-only: guarded to no-op on the SQLite bind (tests / CI / offline
Alembic), where both sinks are inert.
Revision ID: 0004_ui_event_sessions
Revises: 0003_ui_events
Create Date: 2026-07-23
"""
from alembic import op
revision = "0004_ui_event_sessions"
down_revision = "0003_ui_events"
branch_labels = None
depends_on = None
# The behavioural-sequence table. Short on purpose: a funnel question is answered
# in days-to-weeks, and the derived answer should be rolled up into the anonymous
# aggregate rather than kept as raw sequences.
SESSION_RETENTION_DAYS = 30
# Free-text search misses. Long enough to spot a seasonal gap in the place index,
# short enough to be defensible for a field a person can type anything into.
SEARCH_MISS_RETENTION_DAYS = 90
def upgrade() -> None:
if op.get_bind().dialect.name != "postgresql":
return # SQLite fallback (tests/CI): both sinks are no-ops there.
op.execute("CREATE EXTENSION IF NOT EXISTS timescaledb")
# No IP column, no user-agent column, no account-id column, and no unique
# constraint that would require one: there is deliberately nothing here that
# could re-identify a session, and nothing to join it to.
op.execute("""
CREATE TABLE ui_event_session (
ts TIMESTAMPTZ NOT NULL,
session TEXT NOT NULL,
seq INTEGER NOT NULL,
event TEXT NOT NULL,
view TEXT NOT NULL DEFAULT '',
prop TEXT NOT NULL DEFAULT '',
value TEXT NOT NULL DEFAULT '',
referrer TEXT NOT NULL DEFAULT ''
)
""")
# 1-day chunks: the table is dropped a chunk at a time by the retention
# policy, and every query against it is "the last N days".
op.execute(
"SELECT create_hypertable('ui_event_session', by_range('ts', INTERVAL '1 day'))")
op.execute(
f"SELECT add_retention_policy('ui_event_session', "
f"INTERVAL '{SESSION_RETENTION_DAYS} days')")
# The only access pattern is "walk one session in order" (funnels) and
# "everything in a window" (the rollup).
op.execute("CREATE INDEX ui_event_session_sid_idx ON ui_event_session (session, seq)")
# Failed searches: a per-day count, never a row per search.
op.execute("""
CREATE TABLE ui_search_miss (
day DATE NOT NULL,
q TEXT NOT NULL,
count BIGINT NOT NULL DEFAULT 0,
PRIMARY KEY (day, q)
)
""")
# Plain table, not a hypertable: it is tiny (bounded by distinct failed
# queries per day) and the prune is a predicate on `count`, not a chunk drop,
# so time-partitioning buys nothing. Retention is enforced by the same daily
# job that applies the k-anonymity prune -- see core/events.py
# PRUNE_SEARCH_MISS_SQL and UI-EVENTS.md.
op.execute("CREATE INDEX ui_search_miss_day_idx ON ui_search_miss (day)")
def downgrade() -> None:
if op.get_bind().dialect.name != "postgresql":
return
op.execute("DROP TABLE IF EXISTS ui_search_miss")
op.execute("DROP TABLE IF EXISTS ui_event_session")

312
backend/core/events.py
View file

@ -5,45 +5,84 @@ since-start counters. metrics.py answers "is traffic flowing right now"; this
module answers "did anyone ever get a location, and did search work" — over module answers "did anyone ever get a location, and did search work" — over
weeks, across deploys. weeks, across deploys.
**Feature-flagged OFF.** Nothing here records anything unless **Two tiers, two flags, both OFF by default.** Everything here is inert unless
``THERMOGRAPH_EVENTS=1``. With the flag unset the beacon still answers 204 and the matching flag is set, so shipping it is a no-op.
this module is inert, so shipping it is a no-op.
* **Tier A — anonymous aggregates (``THERMOGRAPH_EVENTS``).** Hourly counts keyed
by allowlisted enums. No identifier of any kind, nothing written to the
visitor's device, so it needs no consent. This runs for *every* visitor and is
the unbiased signal.
* **Tier B — consented session rows (``THERMOGRAPH_EVENT_SESSIONS``).** The same
events, additionally carrying an ephemeral per-tab session id, written one row
per event so a funnel ("of the sessions that opened the map, how many reached a
grade?") can actually be computed. **Only ever sent by a browser whose visitor
gave opt-in consent** — see frontend/static/consent.js. This is a behavioural
sequence dataset, which is exactly what Tier A was designed to avoid, so it
carries a much shorter retention and extra minimisation.
Tier B depends on Tier A: with ``THERMOGRAPH_EVENTS`` off, nothing records at all.
Design constraints, in priority order: Design constraints, in priority order:
1. **No identifier, ever.** No cookie, no localStorage id, no fingerprint, no 1. **No stable identifier.** No cookie, no cross-session id, no fingerprint, and
user id — not even when the request carries a logged-in session. The unit of never the account id — see ``validate_session``. The Tier B id is minted in
record is an *hour bucket + a tuple of allowlisted enum values*, never a the browser, lives in ``sessionStorage`` (per tab), rotates on a 30-minute idle
person and never a row-per-interaction. That is a structural guarantee: there / 2-hour absolute cap, and is unlinkable to any other session, device, or
is no column an identifier could be written into. account. Tier A has no id column at all.
2. **No free text, no coordinates.** Every field is drawn from a fixed 2. **No free text, no coordinates.** Every event field is drawn from a fixed
allowlist (see ``SCHEMA``). An unrecognised name or value collapses to a allowlist (see ``SCHEMA``); an unrecognised name or value collapses to a
bounded "other" bucket rather than being stored. So the endpoint cannot be bounded "other" bucket. The one deliberate exception is the failed-search text
turned into arbitrary storage, and cannot accidentally capture a search captured server-side by ``record_search_miss`` — heavily normalised, rejected
query, a URL, or a lat/lon. on any personal-data smell, and pruned unless several people typed the same
3. **No IP.** The caller's IP is used only as an in-memory rate-limit key for thing. It is never accepted from the browser.
the current minute (see ``metrics._rate_ok``) and is never passed in here. 3. **No IP.** The caller's IP is used only as an in-memory rate-limit key for the
current minute (see ``metrics.rate_ok``) and is never passed in here, never
stored, and never joined to a session id.
4. **Never breaks a request.** Every entry point swallows its own errors. 4. **Never breaks a request.** Every entry point swallows its own errors.
Storage: an hourly aggregate upsert into ``ui_event_hourly`` (a TimescaleDB Storage (Postgres; see ``alembic/versions/0003_ui_events.py`` and ``0004``):
hypertable on Postgres — see ``alembic/versions/0003_ui_events.py``), plus one
JSONL line per event into the activity stream so Loki/Grafana can show the last * ``ui_event_hourly`` — Tier A hourly aggregate, 400-day retention.
24h live. Neither carries anything person-linked. Off Postgres (dev, tests, * ``ui_event_session`` — Tier B per-event rows, **30-day** retention, minute-
offline tooling) the aggregate sink degrades to a no-op and only the JSONL line granularity timestamps and an in-session sequence number instead of precise
is written — the same dialect split ``data/store.py`` and ``core/metrics.py`` clock times.
already use. * ``ui_search_miss`` — normalised zero-result queries, 90-day retention, with
singletons pruned at 7 days (see ``SEARCH_MISS_MIN_COUNT``).
Plus one JSONL line per event into the activity stream so Loki/Grafana can show
the live picture. Off Postgres (dev, tests, offline tooling) the SQL sinks
degrade to no-ops and only the JSONL line is written — the same dialect split
``data/store.py`` and ``core/metrics.py`` already use.
""" """
import datetime import datetime
import os import os
import re
import threading import threading
import unicodedata
from core import audit from core import audit
# --- feature flag ------------------------------------------------------------- # --- feature flags -------------------------------------------------------------
# Default OFF everywhere. Turn on per-environment (dev first, then beta, then # Default OFF everywhere. Turn on per-environment (dev first, then beta, then
# prod) once the privacy copy in frontend_ssr/templates/privacy.html.j2 has been # prod) once the privacy copy in frontend/templates/privacy.html.j2 matches what
# updated to match what is actually recorded. # is actually recorded.
ENABLED = os.environ.get("THERMOGRAPH_EVENTS", "").strip().lower() in ("1", "true", "yes", "on")
def _flag(name: str) -> bool:
return os.environ.get(name, "").strip().lower() in ("1", "true", "yes", "on")
#: Tier A. Nothing at all records without this.
ENABLED = _flag("THERMOGRAPH_EVENTS")
#: Tier B. Consented, session-scoped rows. Separate flag on purpose: Tier A is
#: the thing worth running permanently, and Tier B should be switchable off
#: (e.g. at the end of a time-boxed study) without losing the aggregate history.
SESSIONS_ENABLED = _flag("THERMOGRAPH_EVENT_SESSIONS")
#: Zero-result search text. Separate flag again — it is the only free-text field
#: in the system and deserves its own on/off switch and its own decision.
SEARCH_MISS_ENABLED = _flag("THERMOGRAPH_SEARCH_MISS")
DATABASE_URL = os.environ.get("THERMOGRAPH_DATABASE_URL", "").strip() DATABASE_URL = os.environ.get("THERMOGRAPH_DATABASE_URL", "").strip()
IS_POSTGRES = DATABASE_URL.startswith("postgresql") IS_POSTGRES = DATABASE_URL.startswith("postgresql")
@ -165,11 +204,118 @@ SLOTS = ("view", "prop", "value")
MAX_BATCH = 20 MAX_BATCH = 20
#: Max JSON body the beacon will read, in bytes. Must comfortably exceed #: Max JSON body the beacon will read, in bytes. Must comfortably exceed
#: MAX_BATCH full-sized events (~90 bytes each) or the byte cap, not the batch #: MAX_BATCH full-sized events (~120 bytes each with a session id) or the byte
#: cap, becomes the binding limit and a legitimate full batch is silently #: cap, not the batch cap, becomes the binding limit and a legitimate full batch
#: dropped. Still far too small for the endpoint to be worth abusing as storage. #: is silently dropped. Still far too small to be worth abusing as storage.
MAX_BODY_BYTES = 4096 MAX_BODY_BYTES = 4096
# --- Tier B: the ephemeral session id ------------------------------------------
# The id is minted in the browser (16 random bytes, base64url, 22 chars) and is
# the ONE field a client could otherwise use to smuggle arbitrary text into the
# pipeline — a free-text column with a different name. So it is validated by
# shape, not merely length-capped: anything that is not exactly 22 base64url
# characters is discarded and the event still records to Tier A. There is
# nothing to gain by sending a "clever" id, and nothing gets stored if you try.
_SESSION_RE = re.compile(r"\A[A-Za-z0-9_-]{22}\Z")
#: Rows per session before Tier B stops recording (Tier A keeps counting). Bounds
#: both storage and how long/detailed any one behavioural sequence can get — a
#: 200-step sequence is already far more than any funnel question needs.
MAX_SEQ = 200
def validate_session(sid) -> str:
"""A well-formed ephemeral session id, or "" — never anything else.
Deliberately NOT a sanitiser: an id that does not match the expected shape is
dropped entirely rather than trimmed into shape, because the only thing a
malformed id can be is either a bug or an attempt to use the field as a
free-text channel, and both should lose their payload completely.
The account id is never accepted here and is never read on this route: see
web/app.py's api_event, which does not touch the session cookie. Tying an
anonymous behavioural sequence to an identified person would convert this
whole dataset into personal data about a known individual, for essentially
no analytical gain (the questions are about the product, not about who).
"""
if not isinstance(sid, str):
return ""
return sid if _SESSION_RE.match(sid) else ""
def validate_seq(seq) -> int:
"""An in-session sequence number in [0, MAX_SEQ], or -1 for "not in a
session". Ordering is carried by this counter rather than by a precise
timestamp, so the stored time can stay coarse (see ``_minute_bucket``)."""
if isinstance(seq, bool) or not isinstance(seq, int):
return -1
return seq if 0 <= seq <= MAX_SEQ else -1
# --- the one free-text field: zero-result search queries ------------------------
# Captured server-side only (api_suggest), never accepted from a browser. What
# people searched for and did NOT find is the single highest-value product signal
# here — it names the gaps in the place index — but a search box is a field a
# person can type anything into, so this is the only place where personal data
# could arrive incidentally. Four layers of mitigation, in order of strength:
#
# 1. Only ZERO-RESULT queries are considered at all. A query that found a place
# is already answered by the `hit` counter and is never stored as text.
# 2. Reject (never sanitise) anything with a personal-data smell — an address,
# an email, a URL, a long digit run, a sentence. Stripping bad characters
# would keep the text; rejecting loses it, which is the point.
# 3. Store as a COUNT per normalised string per day, not as a row per search.
# 4. Prune anything only ever typed by a couple of people (see
# SEARCH_MISS_MIN_COUNT). A string three different people typed is a product
# signal; a string one person typed once is the risky case, and it is the
# one that gets deleted.
#: Longer than this is a sentence or an address, not a place name.
SEARCH_MISS_MAX_LEN = 40
#: More words than this likewise.
SEARCH_MISS_MAX_WORDS = 5
#: A run of digits this long is a phone number / ID, not a postcode.
SEARCH_MISS_MAX_DIGIT_RUN = 6
#: Rows below this count are deleted once they are older than
#: SEARCH_MISS_PRUNE_DAYS — a k-anonymity floor on the free-text field.
SEARCH_MISS_MIN_COUNT = 3
SEARCH_MISS_PRUNE_DAYS = 7
# Letters (any script), marks, digits, and the punctuation that genuinely occurs
# in place names. Anything else is a smell, not a typo.
_PLACE_OK = re.compile(r"\A[^\W_]+(?:[ \-'.,()·’][^\W_]+)*\Z", re.UNICODE)
_DIGIT_RUN = re.compile(r"\d{%d,}" % (SEARCH_MISS_MAX_DIGIT_RUN + 1))
_URLISH = re.compile(r"(://|www\.|\.[a-z]{2,}(/|\Z))", re.IGNORECASE)
def normalize_search_miss(q) -> str:
"""A failed query reduced to a storable form, or "" to store nothing.
NFKC-normalised, casefolded, whitespace-collapsed, length- and word-capped.
Returns "" — meaning *drop it* — for anything that smells like personal data
rather than a place name.
"""
if not isinstance(q, str):
return ""
q = unicodedata.normalize("NFKC", q).strip()
q = " ".join(q.split()) # collapse all internal whitespace runs
if not q:
return ""
# Casefold, not lower(): correct for ß/İ and friends, and it collapses the
# trivial "same query, different capitalisation" variants into one row.
q = q.casefold()
if len(q) > SEARCH_MISS_MAX_LEN:
return ""
if len(q.split(" ")) > SEARCH_MISS_MAX_WORDS:
return ""
if "@" in q or _URLISH.search(q): # email / URL / hostname
return ""
if _DIGIT_RUN.search(q): # phone number, account id, coords
return ""
if not _PLACE_OK.match(q): # punctuation soup, not a place name
return ""
return q
def normalize(name: str, props: "dict | None" = None) -> "tuple[str, dict[str, str]]": def normalize(name: str, props: "dict | None" = None) -> "tuple[str, dict[str, str]]":
"""An allowlisted ``(event, {view, prop, value})`` pair. """An allowlisted ``(event, {view, prop, value})`` pair.
@ -226,6 +372,19 @@ def _hour_bucket() -> datetime.datetime:
return now.replace(minute=0, second=0, microsecond=0) return now.replace(minute=0, second=0, microsecond=0)
def _minute_bucket() -> datetime.datetime:
"""Tier B timestamps are truncated to the minute. Precise clock times are a
fingerprint (inter-event millisecond gaps are individuating, and they let two
rotated sessions be stitched back together by adjacency); the *order* of a
session's events is what a funnel needs, and that is carried by ``seq``."""
now = datetime.datetime.now(datetime.timezone.utc)
return now.replace(second=0, microsecond=0)
def _utc_day() -> datetime.date:
return datetime.datetime.now(datetime.timezone.utc).date()
_UPSERT = """ _UPSERT = """
INSERT INTO ui_event_hourly (bucket, event, view, prop, value, referrer, count) INSERT INTO ui_event_hourly (bucket, event, view, prop, value, referrer, count)
VALUES (%s, %s, %s, %s, %s, %s, 1) VALUES (%s, %s, %s, %s, %s, %s, 1)
@ -249,10 +408,64 @@ def _upsert(event: str, dims: "dict[str, str]", referrer: str) -> None:
_local.conn = None _local.conn = None
# --- public entry point ------------------------------------------------------- _INSERT_SESSION = """
INSERT INTO ui_event_session (ts, session, seq, event, view, prop, value, referrer)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
"""
def _insert_session(session: str, seq: int, event: str,
dims: "dict[str, str]", referrer: str) -> None:
conn = _conn()
if conn is None:
return
try:
conn.execute(_INSERT_SESSION, (_minute_bucket(), session, seq, event,
dims["view"], dims["prop"], dims["value"], referrer))
except Exception: # noqa: BLE001 - a broken connection retires itself
try:
conn.close()
except Exception: # noqa: BLE001
pass
_local.conn = None
_UPSERT_SEARCH_MISS = """
INSERT INTO ui_search_miss (day, q, count) VALUES (%s, %s, 1)
ON CONFLICT (day, q) DO UPDATE SET count = ui_search_miss.count + 1
"""
def _upsert_search_miss(q: str) -> None:
conn = _conn()
if conn is None:
return
try:
conn.execute(_UPSERT_SEARCH_MISS, (_utc_day(), q))
except Exception: # noqa: BLE001
try:
conn.close()
except Exception: # noqa: BLE001
pass
_local.conn = None
#: Deletes the free-text rows that only one or two people ever produced, once
#: they are old enough that the count has stopped growing. Run daily (see the
#: rollout section of UI-EVENTS.md); parameterised here rather than typed by hand
#: on the box so the retention rule is reviewable in the repo.
PRUNE_SEARCH_MISS_SQL = (
"DELETE FROM ui_search_miss "
"WHERE day < CURRENT_DATE - INTERVAL '%d days' AND count < %d"
% (SEARCH_MISS_PRUNE_DAYS, SEARCH_MISS_MIN_COUNT)
)
# --- public entry points -------------------------------------------------------
def record(name: str, props: "dict | None" = None, def record(name: str, props: "dict | None" = None,
referer: "str | None" = None, host: "str | None" = None) -> None: referer: "str | None" = None, host: "str | None" = None,
session: "str | None" = None, seq=None) -> None:
"""Record one product event. Best-effort, silent, and a no-op unless the """Record one product event. Best-effort, silent, and a no-op unless the
feature flag is on. feature flag is on.
@ -262,6 +475,12 @@ def record(name: str, props: "dict | None" = None,
a search query or a private path. The referrer dimension is kept only for a search query or a private path. The referrer dimension is kept only for
``view.open``; every other event stores "" there, which is what keeps the ``view.open``; every other event stores "" there, which is what keeps the
key space small. key space small.
``session``/``seq`` are the Tier B additions. They are only ever populated by
a browser whose visitor opted in, they are validated by shape (see
``validate_session``), and Tier A records identically whether they are
present or not — so a dropped or refused session id costs the aggregate
nothing.
""" """
if not ENABLED: if not ENABLED:
return return
@ -271,10 +490,37 @@ def record(name: str, props: "dict | None" = None,
from core import metrics from core import metrics
event, dims = normalize(name, props) event, dims = normalize(name, props)
referrer = metrics.normalize_referrer(referer, host) if event == "view.open" else "" referrer = metrics.normalize_referrer(referer, host) if event == "view.open" else ""
# Tier A first and unconditionally: the anonymous aggregate is the signal
# that must never depend on consent rates, client behaviour, or Tier B.
_upsert(event, dims, referrer) _upsert(event, dims, referrer)
sid = validate_session(session) if SESSIONS_ENABLED else ""
n = validate_seq(seq) if sid else -1
if sid and n >= 0:
_insert_session(sid, n, event, dims, referrer)
# Second sink: one structured line per event into the activity stream, so # Second sink: one structured line per event into the activity stream, so
# the Loki/Grafana stack can show the last 24h live without waiting on an # the Loki/Grafana stack can show the live picture without waiting on an
# hourly rollup. Carries the same allowlisted enums and nothing else. # hourly rollup. Carries the same allowlisted enums and nothing else —
# NOT the session id, which would put a behavioural sequence into a log
# stream with its own retention and its own access rules.
audit.log_activity("ui.event", {"event": event, **dims, "referrer": referrer}) audit.log_activity("ui.event", {"event": event, **dims, "referrer": referrer})
except Exception: # noqa: BLE001 - instrumentation must never break a request except Exception: # noqa: BLE001 - instrumentation must never break a request
pass pass
def record_search_miss(q) -> None:
"""Record one zero-result search query, server-side only.
Never call this with a query that returned results, and never accept the
string from a browser: the browser sending its own search text would put it
in a request body, in the client, and in anything that logs bodies — all of
which this deliberately avoids. Silently stores nothing when the query does
not survive ``normalize_search_miss``.
"""
if not (ENABLED and SEARCH_MISS_ENABLED):
return
try:
norm = normalize_search_miss(q)
if norm:
_upsert_search_miss(norm)
except Exception: # noqa: BLE001 - instrumentation must never break a request
pass

130
backend/tests/core/test_events.py
View file

@ -117,3 +117,133 @@ def test_record_never_raises(monkeypatch):
monkeypatch.setattr(m, "_upsert", boom) monkeypatch.setattr(m, "_upsert", boom)
monkeypatch.setattr(m.audit, "log_activity", boom) monkeypatch.setattr(m.audit, "log_activity", boom)
m.record("view.open", {"view": "home"}) # must not propagate m.record("view.open", {"view": "home"}) # must not propagate
# --- Tier B: the ephemeral session id ------------------------------------------
# The session field is the one place a client could put arbitrary text into
# storage, so it is validated by SHAPE and dropped whole on any mismatch. These
# are the tests that keep it from quietly becoming a free-text column.
def test_session_id_accepts_only_the_exact_minted_shape(monkeypatch):
m = _fresh(monkeypatch)
good = "AbCd1234_-efGH5678ijK" # 21 chars: too short
assert m.validate_session(good) == ""
assert m.validate_session("A" * 22) == "A" * 22
assert m.validate_session("aZ0_-" + "b" * 17) == "aZ0_-" + "b" * 17
def test_session_id_rejects_anything_that_could_carry_text(monkeypatch):
m = _fresh(monkeypatch)
for bad in ("user@example.com", "47.6062,-122.3321", "a" * 200, "", None, 12345,
"AAAAAAAAAAAAAAAAAAAA<>", "has space here aaaaaaa", {"id": "x"},
"user-id-42", "AAAAAAAAAAAAAAAAAAAAA;DROP"):
assert m.validate_session(bad) == "", bad
def test_sequence_number_is_bounded(monkeypatch):
m = _fresh(monkeypatch)
assert m.validate_seq(0) == 0
assert m.validate_seq(m.MAX_SEQ) == m.MAX_SEQ
for bad in (-1, m.MAX_SEQ + 1, 10**9, "3", 3.0, True, None):
assert m.validate_seq(bad) == -1, bad
def test_session_rows_need_their_own_flag(monkeypatch):
"""Tier A must record with Tier B off — the anonymous signal never depends on
consent rates or on the session tier being enabled."""
m = _fresh(monkeypatch)
monkeypatch.setattr(m, "SESSIONS_ENABLED", False)
agg, rows = [], []
monkeypatch.setattr(m, "_upsert", lambda *a: agg.append(a))
monkeypatch.setattr(m, "_insert_session", lambda *a: rows.append(a))
monkeypatch.setattr(m.audit, "log_activity", lambda *a: None)
m.record("view.open", {"view": "home"}, session="A" * 22, seq=1)
assert len(agg) == 1 and rows == []
def test_session_row_written_when_consented_and_enabled(monkeypatch):
m = _fresh(monkeypatch)
monkeypatch.setattr(m, "SESSIONS_ENABLED", True)
agg, rows = [], []
monkeypatch.setattr(m, "_upsert", lambda *a: agg.append(a))
monkeypatch.setattr(m, "_insert_session", lambda *a: rows.append(a))
monkeypatch.setattr(m.audit, "log_activity", lambda *a: None)
m.record("place.pick", {"view": "home", "prop": "map"}, session="B" * 22, seq=4)
assert len(agg) == 1
(sid, seq, event, dims, _ref), = rows
assert (sid, seq, event) == ("B" * 22, 4, "place.pick")
assert dims["prop"] == "map"
def test_bad_session_id_does_not_cost_the_aggregate(monkeypatch):
m = _fresh(monkeypatch)
monkeypatch.setattr(m, "SESSIONS_ENABLED", True)
agg, rows = [], []
monkeypatch.setattr(m, "_upsert", lambda *a: agg.append(a))
monkeypatch.setattr(m, "_insert_session", lambda *a: rows.append(a))
monkeypatch.setattr(m.audit, "log_activity", lambda *a: None)
m.record("view.open", {"view": "home"}, session="not-a-valid-id", seq=1)
assert len(agg) == 1 and rows == []
def test_session_id_never_reaches_the_log_stream(monkeypatch):
"""The activity JSONL goes to Loki with its own retention and access rules;
a behavioural sequence must not be duplicated into it."""
m = _fresh(monkeypatch)
monkeypatch.setattr(m, "SESSIONS_ENABLED", True)
lines = []
monkeypatch.setattr(m, "_upsert", lambda *a: None)
monkeypatch.setattr(m, "_insert_session", lambda *a: None)
monkeypatch.setattr(m.audit, "log_activity", lambda tag, rec: lines.append(rec))
m.record("view.open", {"view": "home"}, session="C" * 22, seq=2)
assert "C" * 22 not in str(lines)
assert "session" not in lines[0] and "seq" not in lines[0]
# --- the one free-text field: zero-result search queries ------------------------
def test_search_miss_normalisation(monkeypatch):
m = _fresh(monkeypatch)
assert m.normalize_search_miss(" Saint-Étienne ") == "saint-étienne"
assert m.normalize_search_miss("NEW\tYORK\n CITY") == "new york city"
assert m.normalize_search_miss("Königsberg") == m.normalize_search_miss("KÖNIGSBERG")
# A postcode-length digit run is a legitimate place query and survives.
assert m.normalize_search_miss("98101") == "98101"
def test_search_miss_rejects_anything_with_a_personal_data_smell(monkeypatch):
m = _fresh(monkeypatch)
for bad in (
"someone@example.com", # email
"https://example.com/x", # URL
"www.example.com", # hostname
"call me on 07700900123", # long digit run
"x" * 60, # too long
"my house is the third one on the left", # a sentence
"$$$ ???", # punctuation soup
"47.6062,-122.3321", # coordinates
None, 42, "",
):
assert m.normalize_search_miss(bad) == "", bad
def test_search_miss_needs_its_own_flag(monkeypatch):
m = _fresh(monkeypatch)
monkeypatch.setattr(m, "SEARCH_MISS_ENABLED", False)
seen = []
monkeypatch.setattr(m, "_upsert_search_miss", lambda q: seen.append(q))
m.record_search_miss("atlantis")
assert seen == []
monkeypatch.setattr(m, "SEARCH_MISS_ENABLED", True)
m.record_search_miss("atlantis")
m.record_search_miss("someone@example.com")
assert seen == ["atlantis"]
def test_search_miss_prune_enforces_a_k_anonymity_floor(monkeypatch):
"""The strongest mitigation on the free-text field: a string only one or two
people typed is deleted; one several people typed is a product signal."""
m = _fresh(monkeypatch)
assert m.SEARCH_MISS_MIN_COUNT >= 3
assert f"count < {m.SEARCH_MISS_MIN_COUNT}" in m.PRUNE_SEARCH_MISS_SQL
assert f"{m.SEARCH_MISS_PRUNE_DAYS} days" in m.PRUNE_SEARCH_MISS_SQL

35
backend/tests/web/test_api.py
View file

@ -322,3 +322,38 @@ def test_score_shape_and_conditional_revalidation(score_client):
# Second GET replays the exact same bytes from the derived store. # Second GET replays the exact same bytes from the derived store.
r2 = score_client.get("/thermograph/api/v2/score", params=Q) r2 = score_client.get("/thermograph/api/v2/score", params=Q)
assert r2.status_code == 200 and r2.content == r.content assert r2.status_code == 200 and r2.content == r.content
# --- failed-search capture (server-side only) ----------------------------------
def test_zero_result_search_records_the_query_text_server_side(client, monkeypatch):
"""Decision 2: the miss text is captured in the handler, never sent from the
browser. Only misses — a query that found something is never stored."""
from core import events
seen = []
monkeypatch.setattr(events, "ENABLED", True)
monkeypatch.setattr(events, "SEARCH_MISS_ENABLED", True)
monkeypatch.setattr(events, "_upsert", lambda *a: None)
monkeypatch.setattr(events, "_upsert_search_miss", lambda q: seen.append(q))
monkeypatch.setattr(events.audit, "log_activity", lambda *a: None)
monkeypatch.setattr(places, "suggest", lambda q, up, lim: ([], None))
client.get("/thermograph/api/v2/suggest", params={"q": "Atlantis"})
assert seen == ["atlantis"]
monkeypatch.setattr(places, "suggest",
lambda q, up, lim: ([{"name": "Seattle"}], None))
client.get("/thermograph/api/v2/suggest", params={"q": "Seattle"})
assert seen == ["atlantis"] # a hit is never stored as text
def test_search_miss_text_is_not_written_to_the_access_log(client, monkeypatch):
"""The app's own access log records url.path, never the query string — so the
search text cannot leak into it. (Caddy's log is a separate concern: it logs
the full URI, which is why the miss capture must stay server-side and why the
Caddy log format needs the query stripped -- see UI-EVENTS.md.)"""
from core import audit
lines = []
monkeypatch.setattr(audit, "log_access", lambda rec: lines.append(rec))
monkeypatch.setattr(places, "suggest", lambda q, up, lim: ([], None))
client.get("/thermograph/api/v2/suggest", params={"q": "somewhere-secret"})
assert lines and all("somewhere-secret" not in str(v) for rec in lines for v in rec.values())

53
backend/tests/web/test_event_beacon.py
View file

@ -139,3 +139,56 @@ def test_beacon_is_not_counted_as_inbound_traffic(client):
"""It reports traffic; counting it would double every interaction.""" """It reports traffic; counting it would double every interaction."""
from core import metrics from core import metrics
assert metrics.classify_inbound(EVENT_URL, "/thermograph") == "event" assert metrics.classify_inbound(EVENT_URL, "/thermograph") == "event"
# --- Tier B over the wire -------------------------------------------------------
@pytest.fixture
def session_rows(monkeypatch, recorded):
rows = []
monkeypatch.setattr(events, "SESSIONS_ENABLED", True)
monkeypatch.setattr(events, "_insert_session", lambda *a: rows.append(a))
return rows
def test_session_and_seq_travel_with_a_batch(client, session_rows):
client.post(EVENT_URL, json={
"session": "D" * 22,
"events": [{"event": "view.open", "props": {"view": "home"}, "seq": 1},
{"event": "place.pick", "props": {"view": "home", "prop": "map"}, "seq": 2}],
}, headers=FIRST_PARTY)
assert [(r[0], r[1], r[2]) for r in session_rows] == [
("D" * 22, 1, "view.open"), ("D" * 22, 2, "place.pick")]
def test_a_forged_session_field_cannot_smuggle_text(client, session_rows, recorded):
"""The session field is the only client-controlled string that reaches a
sink, so a malformed one must lose its payload entirely — while the
anonymous aggregate still records."""
for forged in ("'; DROP TABLE ui_event_session; --", "user@example.com",
"47.6062,-122.3321", "x" * 500):
client.post(EVENT_URL, json={"session": forged, "seq": 1,
"event": "view.open", "props": {"view": "home"}},
headers=FIRST_PARTY)
assert session_rows == []
assert len(recorded) == 4 # Tier A recorded every one of them
def test_sessions_flag_off_records_only_the_aggregate(client, monkeypatch, recorded):
rows = []
monkeypatch.setattr(events, "SESSIONS_ENABLED", False)
monkeypatch.setattr(events, "_insert_session", lambda *a: rows.append(a))
client.post(EVENT_URL, json={"session": "E" * 22, "seq": 1,
"event": "view.open", "props": {"view": "home"}},
headers=FIRST_PARTY)
assert rows == [] and len(recorded) == 1
def test_beacon_never_reads_the_account_session_cookie(client, session_rows, recorded):
"""Even a signed-in visitor's events stay unlinked to the account: the route
does not consult the auth cookie, so there is nothing to join on."""
client.cookies.set("thermographauth", "pretend-jwt-value")
client.post(EVENT_URL, json={"event": "view.open", "props": {"view": "home"}},
headers=FIRST_PARTY)
assert "pretend-jwt" not in str(recorded) + str(session_rows)
assert recorded and all("user" not in k for rec in recorded for k in rec)

52
backend/web/app.py
View file

@ -455,14 +455,21 @@ def api_suggest(q: str = Query(..., min_length=1)):
raise HTTPException(status_code=502, detail=f"geocoding failed: {e}") raise HTTPException(status_code=502, detail=f"geocoding failed: {e}")
# "Did search work?" is recorded HERE, not in the browser: the server already # "Did search work?" is recorded HERE, not in the browser: the server already
# knows the answer, every surface that searches is covered for free, and the # knows the answer, every surface that searches is covered for free, and the
# query string never leaves this handler — only the three-way outcome (found / # query string never has to leave this handler. Inert unless
# found-after-respelling / nothing) is stored. Inert unless THERMOGRAPH_EVENTS # THERMOGRAPH_EVENTS is on. NOTE: the type-ahead fires once per debounced
# is on. NOTE: the type-ahead fires once per debounced keystroke, so this # keystroke, so this counts *keystroke batches*, not distinct searches — read
# counts *keystroke batches*, not distinct searches — read the miss RATE, not # the miss RATE, not the miss count.
# the miss count.
events.record("place.search", events.record("place.search",
{"prop": "hit" if results and not corrected {"prop": "hit" if results and not corrected
else "typo" if results else "miss"}) else "typo" if results else "miss"})
if not results:
# The one free-text capture in the system, and server-side only: what
# people looked for and did NOT find names the gaps in the place index.
# events.record_search_miss owns every mitigation — it rejects anything
# with a personal-data smell outright, stores a per-day count rather than
# a row per search, and a daily prune deletes strings only one or two
# people ever typed. Behind its own flag, off by default.
events.record_search_miss(q)
return {"results": results, "corrected": corrected} return {"results": results, "corrected": corrected}
@ -837,16 +844,25 @@ async def api_event(request: Request) -> Response:
metric switch, a dead end the visitor hit, …). metric switch, a dead end the visitor hit, …).
Deliberately minimal: the body carries an allowlisted event name and a few Deliberately minimal: the body carries an allowlisted event name and a few
allowlisted enum properties — no free text, no coordinates, no identifier of allowlisted enum properties — no free text, no coordinates, and **the account
any kind, and the session cookie is deliberately never read. The referrer is session cookie is never read on this route**, so nothing recorded here can be
taken from this request's own Referer header (a client-supplied one would be tied to an identified person even when the visitor is signed in. The referrer
forgeable and buys nothing) and reduced to a bare domain. is taken from this request's own Referer header (a client-supplied one would
be forgeable and buys nothing) and reduced to a bare domain.
Body shapes accepted:: Body shapes accepted::
{"event": "view.open", "props": {"view": "calendar", "prop": "nav"}} {"event": "view.open", "props": {"view": "calendar", "prop": "nav"}}
{"events": [{"event": ..., "props": {...}}, ...]} # batched, <= MAX_BATCH {"events": [{"event": ..., "props": {...}}, ...]} # batched, <= MAX_BATCH
A *consented* client additionally sends ``session`` — an ephemeral, per-tab,
self-rotating id minted in the browser (see frontend/static/consent.js and
track.js) — plus a per-event ``seq``. Both are validated by shape and dropped
whole if they are not exactly what is expected: ``session`` is the one field
a client could otherwise use to smuggle free text into storage. The anonymous
aggregate records identically with or without them, so a refused id costs the
primary signal nothing.
Always answers 204 — counted, rejected by the allowlist, over the body cap, Always answers 204 — counted, rejected by the allowlist, over the body cap,
or dropped by the rate limiter all look identical, so probing the endpoint or dropped by the rate limiter all look identical, so probing the endpoint
reveals nothing. That also suits navigator.sendBeacon, which ignores the reveals nothing. That also suits navigator.sendBeacon, which ignores the
@ -872,26 +888,30 @@ async def api_event(request: Request) -> Response:
# The IP is a rate-limit key for the current minute and nothing else: it is # The IP is a rate-limit key for the current minute and nothing else: it is
# never written to the counters, the aggregate table, or the activity log. # never written to the counters, the aggregate table, or the activity log.
ip = _client_ip(request) ip = _client_ip(request)
# One session id for the whole batch (a batch is by construction one tab's
# events), but each event carries its own sequence number.
session = body.get("session")
for item in batch: for item in batch:
name = str(item.get("event") or "") name = str(item.get("event") or "")
if not name: if not name:
continue continue
props = item.get("props") props = item.get("props")
await run_in_threadpool(_record_event, name, props, referer, host, ip) await run_in_threadpool(_record_event, name, props, referer, host, ip,
session, item.get("seq"))
return Response(status_code=204) return Response(status_code=204)
def _record_event(name, props, referer, host, ip) -> None: def _record_event(name, props, referer, host, ip, session=None, seq=None) -> None:
"""Fan one event out to both tiers (blocking; called via the threadpool). """Fan one event out to every sink (blocking; called via the threadpool).
``metrics`` keeps the since-start live counters the ops endpoint already ``metrics`` keeps the since-start live counters the ops endpoint already
serves — untouched, and still the rate-limit gate. ``events`` adds the serves — untouched, and still the rate-limit gate. ``events`` adds the durable
durable, dimensioned hourly aggregate, and is inert unless THERMOGRAPH_EVENTS anonymous hourly aggregate plus, for a consented client only, the
is on.""" session-scoped row. Both are inert unless their flags are on."""
if not metrics.rate_ok(ip): if not metrics.rate_ok(ip):
return return
metrics.record_event(name, referer=referer, host=host, rate_limit=False) metrics.record_event(name, referer=referer, host=host, rate_limit=False)
events.record(name, props, referer=referer, host=host) events.record(name, props, referer=referer, host=host, session=session, seq=seq)
# --- API versioning -------------------------------------------------------- # --- API versioning --------------------------------------------------------

1
frontend/static/calendar.html
View file

@ -123,6 +123,7 @@
<script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script> <script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script>
<!-- Product-event beacon. Inert unless the server flagged this page <!-- Product-event beacon. Inert unless the server flagged this page
on the root element; see frontend/static/track.js. --> on the root element; see frontend/static/track.js. -->
<script type="module" src="consent.js"></script>
<script type="module" src="track.js"></script> <script type="module" src="track.js"></script>
<script type="module" src="calendar.js"></script> <script type="module" src="calendar.js"></script>
<script type="module" src="ios-install.js"></script> <script type="module" src="ios-install.js"></script>

1
frontend/static/compare.html
View file

@ -179,6 +179,7 @@
<script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script> <script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script>
<!-- Product-event beacon. Inert unless the server flagged this page <!-- Product-event beacon. Inert unless the server flagged this page
on the root element; see frontend/static/track.js. --> on the root element; see frontend/static/track.js. -->
<script type="module" src="consent.js"></script>
<script type="module" src="track.js"></script> <script type="module" src="track.js"></script>
<script type="module" src="compare.js"></script> <script type="module" src="compare.js"></script>
<script type="module" src="ios-install.js"></script> <script type="module" src="ios-install.js"></script>

145
frontend/static/consent.js Normal file
View file

@ -0,0 +1,145 @@
// Analytics consent: the opt-in gate in front of the session-scoped tier.
//
// WHY THIS EXISTS. The anonymous tier (hourly counts, no identifier, nothing
// written to your device) needs no permission. The session tier does: it mints
// an ephemeral id and stores it in sessionStorage, and storing an identifier on
// someone's device engages ePrivacy Art. 5(3). Product analytics is not
// "strictly necessary to provide a service the user explicitly requested", so
// the exemption does not apply and the lawful basis is opt-in consent.
//
// RULES THIS FILE ENFORCES, all of them load-bearing:
// * Nothing is stored and no id is minted before an explicit "Allow". Not on
// first paint, not "pending a choice" — refusing by ignoring the banner must
// leave exactly as little behind as refusing explicitly.
// * Allow and No are the same size, same weight, same prominence. No pre-tick,
// no "legitimate interest" second layer, no cookie-wall.
// * Withdrawal is as easy as granting, reachable at any time from the footer
// and the privacy page, and it deletes the live session id immediately.
// * GPC / DNT are treated as a refusal already given: no banner is shown and
// nothing is stored. Stricter than the letter of the law, and it means the
// most privacy-conscious visitors are never nagged.
//
// The consent RECORD itself lives in localStorage. Storing a user's own choice
// is strictly necessary to honour it, so it is exempt from the consent it
// records — but it is deliberately the only thing kept across sessions.
const KEY = "thermograph:analytics-consent";
//: Bump to re-ask everyone — only when what is collected materially changes.
const POLICY_VERSION = 1;
//: Re-ask a granter after this long. Never re-ask a refuser inside it: a banner
//: that keeps reappearing after "no" is a dark pattern, not a question.
const GRANT_TTL_MS = 183 * 24 * 3600 * 1000; // ~6 months
const DENY_TTL_MS = 365 * 24 * 3600 * 1000; // ~12 months
export const signalledOptOut =
navigator.globalPrivacyControl === true ||
navigator.doNotTrack === "1" ||
window.doNotTrack === "1";
const listeners = [];
let state = null; // "granted" | "denied" | null (not asked)
function read() {
try {
const raw = localStorage.getItem(KEY);
if (!raw) return null;
const rec = JSON.parse(raw);
if (rec.v !== POLICY_VERSION) return null;
const ttl = rec.choice === "granted" ? GRANT_TTL_MS : DENY_TTL_MS;
if (!rec.ts || Date.now() - rec.ts > ttl) return null;
return rec.choice === "granted" ? "granted" : "denied";
} catch { return null; }
}
function write(choice) {
try {
localStorage.setItem(KEY, JSON.stringify({ v: POLICY_VERSION, choice, ts: Date.now() }));
} catch { /* private mode: the choice just won't persist */ }
}
/** "granted" | "denied" | null. GPC/DNT short-circuits to "denied". */
export function consentState() {
if (signalledOptOut) return "denied";
if (state === null) state = read();
return state;
}
export const hasConsent = () => consentState() === "granted";
/** Called with the new state whenever it changes — track.js uses this to drop
* the live session id the instant consent is withdrawn. */
export function onConsentChange(cb) { listeners.push(cb); }
export function setConsent(choice) {
state = choice === "granted" ? "granted" : "denied";
write(state);
hideBanner();
paintControls();
for (const cb of listeners) { try { cb(state); } catch { /* never break */ } }
}
// --- the banner ---------------------------------------------------------------
// Rendered only when instrumentation is enabled AND no choice has been recorded
// AND no opt-out signal is present. It is a bar, not a modal: it never blocks
// reading the page, which is the whole product.
let banner = null;
function hideBanner() {
if (banner) { banner.remove(); banner = null; }
}
function showBanner() {
if (banner) return;
banner = document.createElement("div");
banner.className = "consent-bar";
banner.setAttribute("role", "dialog");
banner.setAttribute("aria-label", "Analytics");
banner.innerHTML = `
<p>Can we count how this page gets used? Anonymous, kept for 30 days, never
shared, and never linked to you or to an account.
<a class="consent-more" href="">What this collects</a></p>
<div class="consent-actions">
<button type="button" class="consent-no">No thanks</button>
<button type="button" class="consent-yes">Allow</button>
</div>`;
// The privacy link is resolved against the page, not hardcoded, so it is right
// under both the root-served and /thermograph-prefixed deployments.
const more = banner.querySelector(".consent-more");
more.href = new URL("privacy", document.baseURI).pathname;
banner.querySelector(".consent-yes").addEventListener("click", () => setConsent("granted"));
banner.querySelector(".consent-no").addEventListener("click", () => setConsent("denied"));
document.body.appendChild(banner);
}
// --- the withdrawal control ---------------------------------------------------
// Any [data-consent-control] element becomes a live "Analytics: on/off — change"
// toggle. The footer carries one on every page, so withdrawing is never more
// than one click from wherever the visitor is.
function paintControls() {
const s = consentState();
for (const el of document.querySelectorAll("[data-consent-control]")) {
if (signalledOptOut) {
el.textContent = "Analytics: off (your browser asked us not to)";
continue;
}
const on = s === "granted";
el.innerHTML =
`Analytics: ${on ? "on" : "off"} <button type="button" class="consent-toggle">`
+ `${on ? "Turn off" : "Turn on"}</button>`;
el.querySelector(".consent-toggle").addEventListener(
"click", () => setConsent(on ? "denied" : "granted"));
}
}
export function initConsent() {
// Instrumentation disabled for this environment: no banner, no control, no
// storage. The flag is the outermost gate on everything.
if (document.documentElement.dataset.tgEvents !== "1") return;
paintControls();
if (signalledOptOut) return; // already refused, at the browser level
if (consentState() === null) showBanner();
}
initConsent();

1
frontend/static/day.html
View file

@ -85,6 +85,7 @@
<script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script> <script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script>
<!-- Product-event beacon. Inert unless the server flagged this page <!-- Product-event beacon. Inert unless the server flagged this page
on the root element; see frontend/static/track.js. --> on the root element; see frontend/static/track.js. -->
<script type="module" src="consent.js"></script>
<script type="module" src="track.js"></script> <script type="module" src="track.js"></script>
<script type="module" src="day.js"></script> <script type="module" src="day.js"></script>
<script type="module" src="ios-install.js"></script> <script type="module" src="ios-install.js"></script>

1
frontend/static/legend.html
View file

@ -117,6 +117,7 @@
</script> </script>
<!-- Product-event beacon. Inert unless the server flagged this page <!-- Product-event beacon. Inert unless the server flagged this page
on the root element; see frontend/static/track.js. --> on the root element; see frontend/static/track.js. -->
<script type="module" src="consent.js"></script>
<script type="module" src="track.js"></script> <script type="module" src="track.js"></script>
<script type="module" src="ios-install.js"></script> <script type="module" src="ios-install.js"></script>
</body> </body>

1
frontend/static/score.html
View file

@ -72,6 +72,7 @@
<script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script> <script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script>
<!-- Product-event beacon. Inert unless the server flagged this page <!-- Product-event beacon. Inert unless the server flagged this page
on the root element; see frontend/static/track.js. --> on the root element; see frontend/static/track.js. -->
<script type="module" src="consent.js"></script>
<script type="module" src="track.js"></script> <script type="module" src="track.js"></script>
<script type="module" src="score.js"></script> <script type="module" src="score.js"></script>
</body> </body>

58
frontend/static/style.css
View file

@ -2096,3 +2096,61 @@ details.hub-country > summary:hover { color: var(--accent); }
font-size: 22px; line-height: 1; cursor: pointer; font-size: 22px; line-height: 1; cursor: pointer;
} }
.ios-a2hs-x:hover { color: var(--text); background: var(--surface-2); } .ios-a2hs-x:hover { color: var(--text); background: var(--surface-2); }
/* --- analytics consent ---------------------------------------------------- */
/* A bar, never a modal: it must not block reading the page, which is the
product. Both buttons carry identical weight — the "no" is not a muted link,
because a refusal that is harder to click than a grant is not a free choice.
Only rendered when instrumentation is enabled and no choice is recorded. */
.consent-bar {
position: fixed;
inset: auto 0 0 0;
z-index: 60;
display: flex;
flex-wrap: wrap;
align-items: center;
justify-content: center;
gap: 10px 20px;
padding: 12px 16px calc(12px + env(safe-area-inset-bottom));
background: var(--surface-2);
border-top: 1px solid var(--border);
box-shadow: 0 -4px 16px rgb(0 0 0 / 0.25);
}
.consent-bar p { margin: 0; max-width: 60ch; font-size: 14px; color: var(--text); }
.consent-bar a { color: var(--accent); }
.consent-actions { display: flex; gap: 10px; }
/* Equal size, equal prominence, both >= the 44px touch target. */
.consent-bar button {
min-height: 44px;
min-width: 116px;
padding: 10px 18px;
font-size: 16px;
font-weight: 600;
border-radius: 8px;
border: 1px solid var(--border);
background: var(--surface);
color: var(--text);
cursor: pointer;
}
.consent-bar .consent-yes { border-color: var(--accent); color: var(--accent); }
.consent-bar button:hover { background: var(--bg); }
/* The always-available withdrawal control in the footer. */
.consent-control { display: block; margin-top: 8px; }
.consent-control .consent-toggle {
min-height: 32px;
padding: 4px 10px;
margin-left: 6px;
font-size: 14px;
border-radius: 6px;
border: 1px solid var(--border);
background: transparent;
color: var(--accent);
cursor: pointer;
}
@media (max-width: 560px) {
.consent-bar { flex-direction: column; align-items: stretch; text-align: left; }
.consent-actions { justify-content: stretch; }
.consent-bar button { flex: 1; }
}

1
frontend/static/subscriptions.html
View file

@ -65,6 +65,7 @@
<script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script> <script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script>
<!-- Product-event beacon. Inert unless the server flagged this page <!-- Product-event beacon. Inert unless the server flagged this page
on the root element; see frontend/static/track.js. --> on the root element; see frontend/static/track.js. -->
<script type="module" src="consent.js"></script>
<script type="module" src="track.js"></script> <script type="module" src="track.js"></script>
<script type="module" src="subscriptions.js"></script> <script type="module" src="subscriptions.js"></script>
<script type="module" src="ios-install.js"></script> <script type="module" src="ios-install.js"></script>

124
frontend/static/track.js
View file

@ -4,32 +4,106 @@
// THERMOGRAPH_EVENTS is on for that environment, so with the flag unset this // THERMOGRAPH_EVENTS is on for that environment, so with the flag unset this
// file costs one dataset read per page and sends nothing at all. // file costs one dataset read per page and sends nothing at all.
// //
// What it will NOT send, by construction: // TWO TIERS, and the difference is the whole privacy story:
// * no identifier of any kind — no cookie, no localStorage id, no fingerprint, //
// no user id even when signed in (sendBeacon cannot attach credentials // Tier A — anonymous. Every visitor, no consent needed, because nothing is
// cross-origin anyway, and the server never reads the session on this route); // stored on the device and nothing identifies anyone. The server keeps hourly
// * no free text — never a search query, never a place name; // counts of allowlisted enums.
//
// Tier B — session-scoped. Adds an ephemeral id so a funnel ("did the visitor
// who opened the picker reach a grade?") can be computed. It requires opt-in
// consent (see consent.js) because it writes an identifier to sessionStorage.
// Without consent, `sid()` returns "" and NOTHING is written to the device —
// Tier A still reports, so refusing costs the product its funnels, not its
// numbers.
//
// What this NEVER sends, in either tier:
// * no stable or cross-site identifier — the Tier B id is per-tab, rotates on
// idle/absolute caps, and is never linked to another session, device, or to
// the signed-in account (sendBeacon cannot attach credentials cross-origin,
// and the server does not read the session cookie on that route);
// * no free text — never a search query, never a place name. Failed searches
// are captured server-side only, never from here;
// * no coordinates — `place.pick` records HOW a location was chosen, not where; // * no coordinates — `place.pick` records HOW a location was chosen, not where;
// * no URLs — the server derives a bare referrer domain from its own headers. // * no URLs — the server derives a bare referrer domain from its own headers.
// Every field is an enum from a fixed allowlist mirrored server-side in // Every field is an enum from a fixed allowlist mirrored server-side in
// backend/core/events.py; anything else is dropped there. // backend/core/events.py; anything else is dropped there.
//
// Global Privacy Control / Do Not Track are honoured: a visitor who has signalled
// either sends nothing. That biases the numbers slightly downward and is worth it.
// Backend's own origin + base + pinned API version, shared with account.js // Backend's own origin + base + pinned API version, shared with account.js
// rather than a second independent copy. // rather than a second independent copy.
import { uv } from "./account.js"; import { uv } from "./account.js";
import { hasConsent, onConsentChange, signalledOptOut } from "./consent.js";
const URL_ = uv("event"); const URL_ = uv("event");
const optedOut = const flagged = () => document.documentElement.dataset.tgEvents === "1";
navigator.globalPrivacyControl === true ||
navigator.doNotTrack === "1" ||
window.doNotTrack === "1";
const enabled = () => // Tier A: the flag, and an explicit browser-level opt-out signal. GPC/DNT are
document.documentElement.dataset.tgEvents === "1" && !optedOut; // honoured here too even though Tier A needs no consent — a visitor who has
// asked not to be measured should not be measured, and the resulting downward
// bias is small and worth it.
const enabled = () => flagged() && !signalledOptOut;
// --- Tier B: the ephemeral session id ------------------------------------------
// sessionStorage, not localStorage and not a cookie:
// * localStorage would persist across sessions and make the id a stable
// pseudonym — the exact thing being avoided;
// * a cookie would ride along on every request including static assets, be
// readable server-side on routes that have nothing to do with analytics,
// and buy nothing here.
// sessionStorage is per-tab and dies with the tab. Consequence, stated honestly:
// a visitor who opens Compare in a second tab is two sessions, so funnels
// UNDERCOUNT completion. That is the error direction to prefer.
const SID_KEY = "thermograph:sid";
const IDLE_MS = 30 * 60 * 1000; // rotate after 30 minutes of inactivity
const MAX_MS = 2 * 60 * 60 * 1000; // and after 2 hours regardless — a hard cap
// on how long any one behavioural sequence
// can possibly be
const MAX_SEQ = 200; // mirrors events.MAX_SEQ server-side
function mintId() {
// 16 random bytes -> 22 base64url chars. Matches the server's _SESSION_RE
// exactly; anything else in this field is dropped whole rather than trimmed.
const b = new Uint8Array(16);
crypto.getRandomValues(b);
return btoa(String.fromCharCode(...b)).replace(/\+/g, "-").replace(/\//g, "_").replace(/=+$/, "");
}
/** The current session id, or "" when there must not be one. Minting is the
* moment something is first written to the device, so it happens ONLY here and
* ONLY behind hasConsent(). */
function sid() {
if (!flagged() || !hasConsent()) return "";
try {
const now = Date.now();
let rec = null;
try { rec = JSON.parse(sessionStorage.getItem(SID_KEY) || "null"); } catch { rec = null; }
if (!rec || !rec.id || now - rec.last > IDLE_MS || now - rec.born > MAX_MS) {
rec = { id: mintId(), born: now, last: now, n: 0 };
}
rec.last = now;
rec.n = Math.min((rec.n || 0) + 1, MAX_SEQ + 1);
sessionStorage.setItem(SID_KEY, JSON.stringify(rec));
return rec.n > MAX_SEQ ? "" : rec.id; // past the cap: back to Tier A only
} catch { return ""; } // private mode / storage disabled: Tier A only
}
function seqOf() {
try {
const rec = JSON.parse(sessionStorage.getItem(SID_KEY) || "null");
return rec && typeof rec.n === "number" ? rec.n : 0;
} catch { return 0; }
}
export function dropSession() {
try { sessionStorage.removeItem(SID_KEY); } catch { /* nothing to drop */ }
}
// Withdrawing consent must take effect immediately, not at the next page load:
// drop the live id and discard anything queued but not yet sent.
onConsentChange((state) => {
if (state !== "granted") { queue = []; dropSession(); }
});
// Batch rather than one request per interaction: chart-metric taps and slider // Batch rather than one request per interaction: chart-metric taps and slider
// drags arrive in bursts, and ~0.03 req/s of real traffic does not need a // drags arrive in bursts, and ~0.03 req/s of real traffic does not need a
@ -40,9 +114,9 @@ const FLUSH_MS = 4000;
let queue = []; let queue = [];
let timer = 0; let timer = 0;
function send(items) { function send(items, session) {
try { try {
const body = JSON.stringify({ events: items }); const body = JSON.stringify(session ? { events: items, session } : { events: items });
if (navigator.sendBeacon) { if (navigator.sendBeacon) {
// sendBeacon survives the page unloading, which a fetch() started on a // sendBeacon survives the page unloading, which a fetch() started on a
// link click usually does not — that's the whole point for nav clicks. // link click usually does not — that's the whole point for nav clicks.
@ -62,7 +136,12 @@ export function flush() {
if (!queue.length) return; if (!queue.length) return;
const items = queue; const items = queue;
queue = []; queue = [];
send(items); // Resolved at flush time, not at queue time: if consent was withdrawn between
// the two, the id is already gone and the batch goes out anonymously. Strip
// the carrier field BEFORE serialising — it must never reach the wire.
const session = hasConsent() ? (items[0].__sid || "") : "";
for (const it of items) delete it.__sid;
send(items, session);
} }
/** Report one product event. `props` may carry only {view, prop, value}, all /** Report one product event. `props` may carry only {view, prop, value}, all
@ -70,7 +149,12 @@ export function flush() {
export function track(event, props) { export function track(event, props) {
try { try {
if (!enabled() || !event) return; if (!enabled() || !event) return;
queue.push(props ? { event, props } : { event }); // sid() both reads and advances the counter, so call it exactly once per
// event. It returns "" (and writes nothing at all) without consent.
const id = sid();
const item = props ? { event, props } : { event };
if (id) { item.seq = seqOf(); item.__sid = id; }
queue.push(item);
if (queue.length >= MAX_BATCH) { flush(); return; } if (queue.length >= MAX_BATCH) { flush(); return; }
if (!timer) timer = setTimeout(flush, FLUSH_MS); if (!timer) timer = setTimeout(flush, FLUSH_MS);
} catch { /* never break the page */ } } catch { /* never break the page */ }
@ -86,8 +170,8 @@ const LEGACY = /^home\.(locate|digest_signup|records_click|share|nav_[a-z]+)$/;
export function trackLegacy(event) { export function trackLegacy(event) {
try { try {
if (optedOut || !event) return; if (signalledOptOut || !event) return;
send([{ event }]); send([{ event }]); // anonymous, unbatched, never session-scoped
} catch { /* never break the page */ } } catch { /* never break the page */ }
} }

8
frontend/templates/base.html.j2
View file

@ -94,7 +94,12 @@
<a href="{{ base }}/privacy">Privacy</a> <a href="{{ base }}/privacy">Privacy</a>
<a href="{{ base }}/">Open the tool</a> <a href="{{ base }}/">Open the tool</a>
</nav> </nav>
<p class="muted">Free. No ads. No tracking. No account needed. Built by one person in the open.</p> <p class="muted">Free. No ads. No third-party trackers. No account needed. Built by one
person in the open.</p>
{# Withdrawing consent must be as easy as giving it and reachable from
anywhere, so the control lives in the site-wide footer. consent.js
fills it in and leaves it empty when instrumentation is disabled. #}
<p class="muted consent-control" data-consent-control></p>
<p class="muted">Thermograph grades weather by its percentile against ~45 years of local <p class="muted">Thermograph grades weather by its percentile against ~45 years of local
climate history. Data: ERA5 via Open-Meteo &middot; NASA POWER &middot; MET Norway.</p> climate history. Data: ERA5 via Open-Meteo &middot; NASA POWER &middot; MET Norway.</p>
<p class="muted">Made by <a href="https://emigriffith.dev">emigriffith.dev</a> <p class="muted">Made by <a href="https://emigriffith.dev">emigriffith.dev</a>
@ -107,6 +112,7 @@
<script type="module" src="{{ asset_base }}/account.js"></script> <script type="module" src="{{ asset_base }}/account.js"></script>
<script type="module" src="{{ asset_base }}/climate.js"></script> <script type="module" src="{{ asset_base }}/climate.js"></script>
{% endblock %} {% endblock %}
<script type="module" src="{{ asset_base }}/consent.js"></script>
<script type="module" src="{{ asset_base }}/digest.js"></script> <script type="module" src="{{ asset_base }}/digest.js"></script>
<script type="module" src="{{ asset_base }}/ios-install.js"></script> <script type="module" src="{{ asset_base }}/ios-install.js"></script>
</body> </body>

2
frontend/templates/home.html.j2
View file

@ -99,7 +99,7 @@
{# Sits outside #panel on purpose: app.js rewrites results.innerHTML {# Sits outside #panel on purpose: app.js rewrites results.innerHTML
wholesale on every grade, so anything inside it would be destroyed. #} wholesale on every grade, so anything inside it would be destroyed. #}
<p class="stance-line muted"> <p class="stance-line muted">
Free. No ads. No tracking. No account needed. Free. No ads. No third-party trackers. No account needed.
<a href="{{ base }}/about">How we work →</a> <a href="{{ base }}/about">How we work →</a>
</p> </p>

68
frontend/templates/privacy.html.j2
View file

@ -16,15 +16,63 @@
learns where you are is if you press <b>Use my location</b>, which asks your browser for learns where you are is if you press <b>Use my location</b>, which asks your browser for
permission. You can decline, and picking a place on the map works just as well.</p> permission. You can decline, and picking a place on the map works just as well.</p>
<p>Whichever way you choose a place, the coordinates are sent to the server only as part of <p>Whichever way you choose a place, the coordinates are sent to the server only as part of
the ordinary request for that grid square's climate data, and are not logged beyond the the ordinary request for that grid square's climate data. The place you picked is
normal web-server request handling every site does. The place you picked is remembered in remembered in your own browser's local storage, not on the server, so clearing your
your own browser's local storage, not on the server, so clearing your browser data browser data forgets it. <b>The coordinates are never recorded in the usage statistics
forgets it.</p> below</b> — those record only <i>how</i> you picked a place (map, search, or your
browser's location), never <i>where</i>.</p>
<h2>Analytics</h2> <h2>Usage statistics</h2>
<p>There is no analytics library, no cookies for tracking, and no per-visitor identifier. <p>There is no third-party analytics service, no advertising tag, and nothing that follows
The server keeps simple aggregate counts (how many people opened a page or tapped a you to other sites. What the site does keep is in two parts, and only the second one
button, and which site referred them) with nothing tying any of it to an individual.</p> asks your permission.</p>
<h3>Anonymous counts — always on</h3>
<p>The server keeps simple counts of things like "the calendar was opened 40 times this
hour" or "a search found nothing 12 times today", along with which site referred the
visit (the domain only, never the full link). No identifier of any kind is attached,
nothing is stored on your device, and there is no way to tell one visitor's counts from
another's — so there is nothing here that could be traced back to you. These counts are
kept for up to 400 days.</p>
<h3>Session statistics — only if you say yes</h3>
<p>To answer questions like "do people who open the location picker actually get a
forecast, or give up half way?", the site has to see several actions as belonging to the
same visit. That means putting a random identifier in your browser, so
<b>we ask first, and it stays off unless you agree</b>.</p>
<p>If you agree:</p>
<ul>
<li>a random number is generated and kept in your browser's <b>session storage</b>, for
the current tab only. It is not a cookie, and it is not sent to any other site;</li>
<li>it changes after 30 minutes of inactivity, and after 2 hours regardless, so it cannot
follow you across a day, a device, or a return visit;</li>
<li>it disappears when you close the tab;</li>
<li>it records which views and controls you used, from a short fixed list — never what you
typed, never where you are, never which articles you read;</li>
<li>it is <b>never linked to your account</b>, even when you are signed in, and never
linked to your email address or your IP address;</li>
<li>the records are deleted after <b>30 days</b>.</li>
</ul>
<p>You can change your mind at any time with the <b>Analytics</b> control at the bottom of
every page. Turning it off deletes the identifier from your browser immediately and stops
any further recording. Because the records are deliberately not linked to you, we cannot
go back and find your past ones to delete — which is part of why they are kept for such a
short time.</p>
<p>If your browser sends a <b>Global Privacy Control</b> or <b>Do Not Track</b> signal, the
site treats that as a no, does not ask, and records nothing extra.</p>
<h3>Searches that found nothing</h3>
<p>When you search for a place and the site finds no match, the text you typed may be kept
so that missing places can be added. It is stored on its own, with nothing else about you
attached, and only if it looks like a place name — anything containing an email address,
a web address or a long run of digits is discarded rather than stored. Anything only one
or two people ever searched for is deleted within a week; the rest is deleted after
90 days. Searches that <i>did</i> find a place are never stored as text.</p>
<h2>Server logs</h2>
<p>As on every website, the server records that a request happened — the page, the time and
the result. Network addresses are shortened before they are written down, so an individual
device cannot be picked out of the logs, and the logs are deleted after 30 days.</p>
<h2>Email</h2> <h2>Email</h2>
<p>If you sign up for the monthly digest, your address is used for that digest and nothing <p>If you sign up for the monthly digest, your address is used for that digest and nothing
@ -32,8 +80,10 @@
<h2>Accounts</h2> <h2>Accounts</h2>
<p>An account is optional and only exists to hold weather alerts you have asked for. It <p>An account is optional and only exists to hold weather alerts you have asked for. It
stores your email address and the places you are watching.</p> stores your email address and the places you are watching. None of the usage statistics
above are connected to it.</p>
<p class="muted">Questions: see <a href="{{ base }}/about">About &amp; methodology</a>.</p> <p class="muted">Questions: see <a href="{{ base }}/about">About &amp; methodology</a>.</p>
<p class="muted consent-control" data-consent-control></p>
</article> </article>
{% endblock %} {% endblock %}

47
frontend/tests/unit/test_pages.py
View file

@ -66,3 +66,50 @@ def test_event_flag_stamps_both_html_paths_when_enabled(monkeypatch):
monkeypatch.delenv("THERMOGRAPH_EVENTS") monkeypatch.delenv("THERMOGRAPH_EVENTS")
importlib.reload(content_mod) importlib.reload(content_mod)
importlib.reload(app_mod) importlib.reload(app_mod)
def test_consent_control_is_present_on_every_page(client):
"""Withdrawal must be reachable from anywhere, so the footer carries the
control on every server-rendered page. consent.js leaves it empty when
instrumentation is disabled, so it costs nothing when off."""
for path in ("/", "/privacy", "/climate"):
r = client.get(f"{B}{path}")
assert r.status_code == 200, path
assert "data-consent-control" in r.text, path
def test_consent_script_loads_before_the_tracker(client):
"""consent.js owns the banner and the opt-in gate; track.js imports it. It is
listed first so the gate is installed even on a page whose tracker chain
never loads."""
for path, after in (("/", "digest.js"), ("/calendar", "track.js")):
body = client.get(f"{B}{path}").text
tag = lambda name: body.index(f'src="{B}/{name}"' if path == "/" else f'src="{name}"')
assert tag("consent.js") < tag(after), path
def test_privacy_page_describes_what_is_actually_collected(client):
"""The page is a public commitment; if this test has to change, the copy and
the code have drifted apart. Asserts the load-bearing promises only."""
body = client.get(f"{B}/privacy").text
for promise in (
"session storage", # where the id lives
"30 minutes", # idle rotation
"2 hours", # absolute cap
"never linked to your account",
"30 days", # session-row retention
"400 days", # anonymous aggregate retention
"90 days", # failed-search retention
"Global Privacy Control",
"only if you say yes", # opt-in, not opt-out
):
assert promise in body, promise
# The v1 claim that this change makes untrue must be gone.
assert "no per-visitor identifier" not in body
def test_footer_no_longer_claims_no_tracking(client):
"""A consented session id is an identifier; the blanket claim would be a lie."""
body = client.get(f"{B}/").text
assert "No tracking." not in body
assert "No third-party trackers." in body