thermograph/UI-EVENTS.md

UI product-event instrumentation — design (v2)

Status: design + prototype. Every tier feature-flagged OFF. Not deployed anywhere. Enabling any of it requires the decisions in §6 and the privacy-page copy in §7 to ship first.

The goal is to learn what people actually do in the Thermograph UI — did they ever get a location, does search work, which views earn their maintenance.

v2 splits the answer into two tiers, because the operator's decisions changed what is being built:

• Tier A — anonymous aggregates. No identifier, nothing stored on the device, no consent needed, runs for every visitor. This was the whole of v1 and is still the primary signal.
• Tier B — consented session rows. An ephemeral per-tab identifier so funnels can be computed. Opt-in consent, short retention, its own flag.

Keeping them separate is the single most important structural choice in v2: if consent rates are poor (they usually are), Tier A still answers most of §1, and Tier B's numbers can be calibrated against Tier A to measure the consent bias rather than pretending it isn't there.

---

What changed from v1, and why

#	v1	v2	Why
1	No identifier at all. Counts of interactions, never of people.	Ephemeral per-tab session id, opt-in, 30 min idle / 2 h absolute rotation, sessionStorage. Tier A unchanged and still runs for everyone.	Operator decision 1: rates can't answer "did this visitor succeed".
2	No consent banner needed (nothing stored on the device).	Opt-in consent banner required for Tier B, with equal-weight Allow/No, one-click withdrawal in the footer of every page, GPC/DNT treated as a refusal already given.	Storing an identifier engages ePrivacy Art. 5(3); analytics is not "strictly necessary". §5.1 argues the exemption case and rejects it.
3	ui_event_hourly aggregate only — deliberately incapable of holding a behavioural sequence.	Adds ui_event_session (one row per event), 30-day retention, minute-granularity timestamps, seq for ordering, 200 events/session cap.	An aggregate cannot express a funnel. This is the sequence dataset v1 avoided, so it gets the shortest retention in the system.
4	Failed search text never captured (§6.2 deferred it).	Captured server-side in api_suggest: normalised, rejected on any personal-data smell, stored as a per-day count, k-anonymity prune at 3.	Operator decision 2.
5	Raw IP in audit.log_access flagged as an open question.	Treated as decided: truncate. Not implemented here (logging-pipeline work owns it); §5.4 states what the app side must do so the two changes stay consistent.	Operator decision 3.
6	Privacy-page rewrite listed as a follow-up.	Written and included (frontend/templates/privacy.html.j2), with tests asserting the load-bearing promises.	The v1 page becomes untrue the moment Tier B ships.
7	One flag, THERMOGRAPH_EVENTS.	Three: THERMOGRAPH_EVENTS (Tier A), THERMOGRAPH_EVENT_SESSIONS (Tier B), THERMOGRAPH_SEARCH_MISS (free text).	Each is a separate decision with a separate risk profile and should be separately revocable.

Unchanged from v1: the seven events and their allowlists, the beacon transport, the abuse-resistance layers, the server-side-derivation preference, and the rejected storage alternatives.

---

0. What already exists (build on this, don't reinvent)

Seam	What it does	Where
Request middleware	classifies + counts every inbound request, writes an access JSONL line including the raw client IP	backend/web/app.py revalidate_static
core/metrics.py	since-start counters (inbound by category, outbound by source, heartbeats) behind GET /api/v2/metrics	backend/core/metrics.py
Product-event beacon	POST /api/v2/event, navigator.sendBeacon, allowlisted names, per-IP rate limit, referrer reduced to a bare domain, always 204	api_event + metrics.record_event
Event counters	(event, referrer_domain, UTC day) → count, 7-day window, UNLOGGED, TRUNCATEd every deploy	metrics.EVENTS, event_cume
Client tracker	track(event) + [data-event] click delegation	frontend/static/digest.js
Activity JSONL	audit.log_activity(tag, record) → logs/activity/*.jsonl	backend/core/audit.py
Loki/Grafana	Alloy ships /applogs/**/*.jsonl fleet-wide, 30-day retention	observability/

So four coarse events (home.locate, home.digest_signup, home.records_click, home.share, plus home.nav_*) are already live, on the SSR homepage only.

Three gaps this design closes.

1. No dimensions. home.locate is a bare counter. It cannot say whether the tap succeeded, which is the only interesting part.
2. No durability. The counters are UNLOGGED and TRUNCATEd on every deploy, and capped at 7 days. Loki holds 30 days of JSONL. Neither survives a quarter, so no seasonal or before/after comparison is possible — which is most of the value for a site about weather.
3. Nothing outside the homepage. Calendar, Day, Compare, Score, Alerts — the whole interactive product — is uninstrumented.

Two corrections to assumptions that were made when this work was scoped: Loki retention is 30 days (observability/loki/config.yml, 720h), not 24h; and the Alloy glob /applogs/**/*.jsonl already picks up the activity/ stream, so a new JSONL sink needs no shipping config.

A third, found while designing v2 and material to decision 2: the log block in infra/deploy/Caddyfile sets no format, so Caddy's default JSON logger records request.uri — the full URI including the query string. Every /api/v2/suggest?q=… a visitor types is therefore already in Loki for 30 days, next to their IP. See §5.4.

---

1. The events worth capturing

Traffic is ~9,700 req/day of which ~60% is crawlers, so ~3,000 human req/day. At that volume a firehose buys nothing a well-chosen aggregate doesn't, and every extra event is another thing to keep honest. Seven events, each tied to a decision someone would actually make:

Event	Question it answers	Decision it informs
view.open	How many sessions reach each view, and from where?	The denominator for everything else; whether Compare/Score justify their maintenance
place.pick	Does anyone ever get past "no place picked", and by which route (geolocate / map / search / restore / link)?	The activation question. If geolocate is mostly declined, promote the map picker
place.search	Does location search find anything? (hit / typo / miss)	Whether to invest in the geocoder/typo tolerance
view.control	Which in-view controls get used — chart metric, °F/°C, date, range, season, compare add/remove, distribution metric	Which chart metrics to keep/promote; whether the unit default is wrong
share	Is the permalink / PNG export used at all?	Whether to keep maintaining the SVG→PNG exporter
alert	The alerts funnel: open → start → create, push on/off	Where subscription signup leaks
deadend	Errors the visitor actually saw: grade error, no data, geolocation denied, search miss, 404, offline	Bug reports nobody filed

Deliberately not captured — because the request log already answers them, and duplicating a signal means two numbers that will eventually disagree: page-view counts, API latency/error rates, and asset traffic. Also deliberately not captured: hover/scroll/dwell (individuating, and answers nothing on this list), map pan/zoom (very high volume, low decision value, and pan/zoom coordinates are location data), and anything at all about which place was chosen.

v2 note: the event list is unchanged. Tier B adds a session id to these same seven events; it does not add an eighth. That is deliberate — the consent conversation is about linking what was already being counted, not about collecting more, and it keeps the privacy-page copy describing one list.

Two events are recorded server-side, not in the browser

place.search is recorded in api_suggest: the server already knows the answer, every surface that searches is covered for free, and the query string never has to leave the request handler — only the three-way outcome is stored, plus (v2, decision 2) the normalised text of the misses only. Same argument would apply to alert creation (accounts/api_accounts.py). Prefer this whenever the server can derive the fact: less client code, no forgeability, and strictly less data in flight.

---

2. Schema

Small, typed, and closed. Three dimension slots (view, prop, value) plus a referrer domain on view.open only. Every value is an enum from a fixed allowlist (backend/core/events.py SCHEMA). Unknown event name → one other bucket with blank dimensions; unknown value in a declared slot → other; a slot an event doesn't declare → dropped entirely. Nothing a client sends is ever persisted verbatim.

view.open     view=<VIEWS>       prop=direct|nav|link|restore    +referrer domain
place.pick    view=<VIEWS>       prop=geolocate|map|search|restore|link
place.search                     prop=hit|typo|miss
view.control  view=<VIEWS>       prop=<CONTROLS>                 value=<VALUES>
share         view=<VIEWS>       prop=link|png|permalink
alert                            prop=open|start|create|push_on|push_off|test  value=y|n
deadend       view=<VIEWS>       prop=grade_error|no_data|geo_denied|search_miss|not_found|offline

VIEWS = home, calendar, day, compare, score, alerts, legend, city, cities, records, month, glossary, about, privacy. CONTROLS = chart_metric, unit, date, range, season, totals, dist_metric, comfort, compare_add, compare_remove, filter_sheet, day_step. VALUES = the metric keys, F/C, today/past/future, count buckets (1…5plus), span buckets (1y…max), season keys, on/off, other.

Numbers are bucketed client-side before they are sent (5plus, not 7): a raw count is a small amount of entropy, and a bucket is all the decision needs.

Tier A storage row: (hour_bucket, event, view, prop, value, referrer) → count. No row per interaction, no column an identifier could go in. Worst-case key space is a few hundred rows/hour; realistically ~50.

Tier B storage row: (minute_ts, session, seq, event, view, prop, value, referrer) — one row per event, written only when the visitor consented. The schema above is unchanged; the session id and sequence number are the only additions, and no new dimension is collected because of Tier B.

The Tier B identifier, specified exactly

Value	16 bytes from crypto.getRandomValues, base64url, 22 chars. Server-validated by exact shape (^[A-Za-z0-9_-]{22}$) — see below.
Where it lives	sessionStorage, key thermograph:sid. Not localStorage (would make it a stable pseudonym across visits — the exact thing being avoided). Not a cookie (would ride on every request including static assets, be readable on routes unrelated to analytics, and buy nothing).
Lifetime	Rotates after 30 min idle; rotates after 2 h absolute regardless of activity; dies with the tab.
Scope	One tab. Two tabs = two sessions; a return visit = a new session; a second device = unrelated. No cross-session, cross-tab, or cross-device linkage exists to break — there is nothing stored that could join them.
Cap	200 events per session, then Tier B stops and Tier A carries on. Bounds both storage and how detailed any one sequence can get.
Minted	Only inside hasConsent(). Refusing, ignoring the banner, or signalling GPC/DNT writes nothing at all to the device.
Account linkage	Never. Argued below.
Retention	30 days, then the chunk is dropped.

Why the session id is never joined to the account id (the decision said "unless you argue otherwise" — the argument is for never):

1. Joining converts an anonymous behavioural sequence into personal data about an identified person, which attaches subject-access, rectification and erasure duties to the entire event history and makes the 30-day retention a floor rather than a nicety.
2. The analytical gain is close to zero: every question in §1 is about the product ("does the picker convert?"), not about who.
3. Signed-in visitors are a small minority, so an account-linked subset is a biased sample you could not generalise from anyway — it would be the most engaged users, i.e. exactly the ones whose funnel is least representative.
4. It is enforced structurally, not by policy: api_event never reads the auth cookie, sendBeacon cannot attach credentials cross-origin, and there is no user column in ui_event_session. There is a test for it.

The session field is the only client-controlled string that reaches storage, so it is validated by shape and dropped whole on any mismatch rather than sanitised into shape. A trimmed-to-fit id would still be attacker-chosen text; a rejected one is nothing. Tier A records identically either way, so refusing a malformed id costs the primary signal zero.

Migration note: the four live home.* events keep their current unflagged, un-dimensioned behaviour so the ops endpoint's numbers don't reset when this lands. Fold them into view.open/place.pick/share and retire the old names as a follow-up, once there's overlap data to stitch the series.

---

3. Transport and storage

Chosen: the existing beacon → two sinks.

• Transport — extend POST /api/v2/event (navigator.sendBeacon, batched, always 204). Already built, already excluded from the traffic counters, already rate-limited. Batching (≤20 events, flushed on a 4 s timer and unconditionally on pagehide/visibilitychange) matters because chart-metric taps and slider drags arrive in bursts.
• Durable sink — ui_event_hourly, a TimescaleDB hypertable (backend/alembic/versions/0003_ui_events.py): 30-day chunks, a native retention policy, and an ON CONFLICT … count + 1 upsert. The DB is already TimescaleDB with hypertables and a compression/retention pattern to copy.
• Live sink — one audit.log_activity("ui.event", …) JSONL line per event. Alloy already ships that path, so the last 30 days are queryable in Grafana immediately without waiting on the rollup. Same allowlisted enums, nothing else.
• Tier B sink — ui_event_session, one row per event, 1-day chunks, 30-day retention policy. Timestamps truncated to the minute and ordering carried by an in-session seq instead: precise clock times are a fingerprint (millisecond inter-event gaps individuate, and adjacency lets two rotated sessions be stitched back together), and a funnel needs order, not timing. No IP column, no user-agent column, no account column, and no unique constraint that would require one.
• Rollup — at 30 days, per-event rows are aggregated into anonymous session-outcome rows (day, entry_view, reached_pick, reached_grade, n) with no identifier, retained alongside Tier A. The sequence data expires; the learning does not. (Specified here, not yet implemented — see §7.)
• The existing in-process counters stay untouched — they remain the live "what's happening right now" view on /api/v2/metrics.

Why not the alternatives:

• Loki only. 30-day retention loses every seasonal comparison — the thing a weather product most wants. Loki is also a poor aggregation engine for "unit switches per week over a year". Kept as the live tier, rejected as the system of record.
• Extend the in-process counters. Cheapest option and no new storage, but they are UNLOGGED, TRUNCATEd every deploy, capped at 7 days, and have exactly one dimension. Every question in §1 needs at least two. Kept, not extended.
• Per-event rows for everyone (a real firehose). Still rejected. v2 adds per-event rows, but only for the consented minority and only for 30 days. Making them the default would put every visitor's behavioural sequence in the database to answer questions that Tier A's aggregate already answers, and would drag the whole dataset behind the consent banner.
• Session-aggregate instead of per-event rows for Tier B (one row per session: entry_view, reached_pick, reached_grade, n_controls). Genuinely tempting — it is strictly less data and answers the headline funnel. Rejected for the live table because the client would have to decide when a session is "done" (it never reliably knows — tabs are closed, not ended) and because a fixed set of outcome columns forecloses any funnel question not thought of in advance. It is adopted as the rollup: at 30 days the per-event rows are aggregated into exactly this shape, with no session id, and kept 400 days — so the sequence data is short-lived and the learning is permanent.
• A third-party analytics SaaS (Plausible/Umami/GA). GA is a non-starter (US transfer, consent banner, cookies). Self-hosted Plausible/Umami would be defensible, but it means another service to run and its own identifier model (Plausible hashes IP+UA+salt — a pseudonymous identifier, which is precisely the decision being deferred to the operator in §6.1). Rejected as more moving parts for less control.

---

4. Abuse and bot resistance

The endpoint is public and unauthenticated on a site where 60% of traffic is crawlers and scanners. Layered, in order of how much they actually buy:

1. Closed allowlist (the real defence). Nothing outside SCHEMA is stored; unknown names collapse into one other bucket. Storage is bounded by the product of the enums, not by what anyone sends. The endpoint cannot be grown into a spam sink or a data-exfil channel.

2. Body cap, enforced while streaming — MAX_BODY_BYTES = 4096; a declared Content-Length past it is refused without reading, and an undeclared one stops mid-stream. (The cap must comfortably exceed a full 20-event batch, or the byte cap silently eats legitimate traffic instead of abusive traffic — there is a regression test for exactly that.)

3. Per-IP rate limit — the existing token bucket, raised from 30/min to 120/min (THERMOGRAPH_EVENT_RATE_PER_MIN) because 30 was sized for four coarse events and a batched interaction stream would silently truncate normal use, biased toward the most engaged visitors. The IP is a key in a map cleared every minute; it is never written anywhere.

4. Soft first-party check — Sec-Fetch-Site must be same-origin/ same-site/none; a cross-site value is dropped, and a POST with neither Sec-Fetch-Site nor Origin is flagged as the first bucket to look at. Every browser that can run our JS sets this; naive scanners don't. Explicitly not a security boundary — anything that bothers can spoof it.

5. Uniform 204 — counted, rejected, over-cap, and rate-limited are indistinguishable, so probing tells an attacker nothing.

6. No UA-based bot filtering. Blocking on User-Agent is a coin flip and gives false confidence. The structural defence is that the beacon requires JS execution and is linked from nowhere.

7. Session-field shape validation (new in v2). session is the one client-controlled string that reaches storage. It is matched against ^[A-Za-z0-9_-]{22}$ and dropped whole on mismatch — never trimmed into shape, because trimmed attacker text is still attacker text. A forged id also cannot inflate anything: Tier A is keyed on enums, not on sessions.

8. Unbounded session minting is the one new abuse surface — an attacker can generate as many valid-looking ids as they like and manufacture fake "sessions". The per-IP rate limit bounds it to 120 rows/min/IP, the 200-event cap bounds any single session, and the 30-day retention bounds the blast radius. The detection signal is the same cross-check as below: Tier B's session count should stay in a stable ratio to Tier A's view.open count, and a sudden divergence means manufactured sessions.

Poisoning (someone inflating a number rather than exhausting storage) cannot be fully prevented without identity. Mitigation is cross-checking: the server's own inbound page counts are an independent measure of view.open, so a divergence between them is the alarm. Any analysis should treat these numbers as directional, never as revenue-grade.

---

5. Privacy position

The site is EU-hosted with EU visitors, so GDPR + ePrivacy apply. The public privacy page promised, verbatim: "no analytics library, no cookies for tracking, and no per-visitor identifier… simple aggregate counts." Tier B makes that false, which is why the rewritten copy (§7) is part of this change and not a follow-up.

5.1 Consent — the argument, and why I land on "banner"

The question is whether Tier B can run without opt-in. It cannot, and here is the reasoning rather than an assumption.

ePrivacy Art. 5(3) is the binding constraint, not GDPR. It covers any storing of, or access to, information on terminal equipment — not just cookies, so sessionStorage is squarely in scope. It admits exactly two exemptions: transmission-necessary, and strictly necessary for a service explicitly requested by the user. Nobody visits a weather site to be measured, so the second exemption does not fit on its plain reading. Crucially, legitimate interest is not available here at all — it is a GDPR Art. 6 lawful basis, and it cannot substitute for the Art. 5(3) consent requirement. Any argument of the form "we have a legitimate interest in analytics, so no banner" is simply answering the wrong question.

The one genuinely arguable route is the audience-measurement exemption: CNIL's guidance (and similar positions from a few other DPAs) exempts a first-party measurement identifier where it is used solely for aggregate statistics, is not shared, is not cross-site, is not combined with other processing, and has bounded lifetimes. This design would fit that description comfortably — arguably better than the tools the exemption was written for, since the id here dies in 2 hours rather than 13 months. So the argument is real, and I am not dismissing it.

Why I still recommend the banner:

1. It is a national regulator's exemption, not harmonised EU law. Germany's TDDDG §25 is read more narrowly, and the EDPB has never blessed a general analytics exemption. EU visitors are not all French.
2. The exemption is conditioned on use — "solely for aggregate statistics". Tier B exists precisely to look at sequences within a session. That is still aggregate at the reporting layer, but it is a harder story to tell than a pageview counter, and the conditions are assessed on what you actually do.
3. The downside is asymmetric. A banner costs a UX annoyance and some sample bias. Getting the exemption wrong on a site with no legal budget costs correspondence with a regulator.
4. Tier A makes the banner cheap. Because the anonymous tier runs regardless, a refusal costs the funnels, not the numbers. If Tier A were also behind the banner I would push much harder on the exemption.

The consent mechanism (frontend/static/consent.js), designed to be a real choice rather than a compliance prop:

• Nothing is stored and no id is minted before an explicit "Allow" — ignoring the banner leaves exactly as little behind as refusing it. There is no "pending" state that quietly collects.
• "Allow" and "No thanks" are the same size, weight and prominence. No pre-ticked boxes, no second-layer "legitimate interest" tab, no cookie-wall.
• It is a bar, not a modal: it never blocks reading the page, which is the product.
• Withdrawal is as easy as granting — an Analytics: on/off control in the footer of every page and on the privacy page. Withdrawing deletes the live session id immediately, discards anything queued-but-unsent, and stops further recording within the same page load.
• Re-asking: a grant is re-confirmed after ~6 months; a refusal is not re-asked for 12 months. A banner that reappears after "no" is a dark pattern, not a question.
• GPC / DNT are treated as a refusal already given: no banner is shown, no id is minted, and Tier A also stands down. Stricter than the law requires (GPC has no binding EU status), and it means the most privacy-conscious visitors are never nagged.
• The consent record itself lives in localStorage. Storing a user's own choice is strictly necessary to honour it, so it is exempt from the consent it records — and it is deliberately the only thing that persists across sessions.

5.2 What is collected, tier by tier

	Tier A (anonymous)	Tier B (consented)
Consent	not required	opt-in required
Identifier	none	ephemeral, per-tab, 30 min idle / 2 h absolute
On-device storage	none	one sessionStorage key
Stored per interaction	nothing (hourly counts)	one row
IP	never	never
Account linkage	never	never
Free text	never	never
Coordinates	never	never
Retention	400 days	30 days, then anonymised rollup
Runs for	everyone	consenting visitors only

5.3 Consequences to be honest about

• The consented sample is self-selected, and the bias is not random. People who decline analytics skew technical and privacy-conscious — on a site about climate percentiles, that is plausibly the same population as the power users whose funnels are most interesting. Tier B may systematically under-represent exactly the behaviour it was added to study.
• Mitigation: every Tier B rate has a Tier A counterpart computed over everyone. Comparing them measures the consent bias directly. If place.pick / view.open differs materially between the two, Tier B's funnels should be read as a lower bound on a subpopulation, not as the truth.
• We cannot honour an erasure request for past events, because the whole point is that nothing links a session to a person — there is no lookup key. The mitigation is the 30-day retention, and the privacy page says so plainly rather than implying a deletion path that does not exist. (A "forget this session" beacon carrying the live id would make it possible for the current session only — specified in §6.6, not built.)
• Two tabs are two sessions, so funnels undercount completion. That is the error direction to prefer, but it must not be forgotten when reading a number.

5.4 IP addresses (decision 3 — not implemented here)

The logging-pipeline work owns this. Recorded here so the two changes stay consistent:

• The app side must stop writing raw IPs. audit.log_access currently writes _client_ip(request) verbatim into logs/access/*.jsonl, which Alloy ships to Loki with 30-day retention. It should write a truncated form — /24 for IPv4, /48 for IPv6 — or drop the field. Truncation preserves the geo-scale and abuse-pattern uses while stopping single-device identification.
• The rate limiter must keep the full IP in memory. metrics.rate_ok needs per-address granularity; truncating its key to /24 would let one abuser rate-limit an entire NAT'd office. Full precision in a map cleared every minute, never written to disk, is the right split.
• Nothing in this design assumes durable raw IPs. No event carries one, and no analysis depends on one. The only cost of truncation is weaker retrospective forensics if the beacon is ever abused — accepted.
• Caddy is the bigger leak, and it is in scope for decision 2. The log block in infra/deploy/Caddyfile sets no format, so it uses Caddy's default JSON logger — which records request.uri, i.e. the full URI including the query string. Every /api/v2/suggest?q=<what someone typed> is therefore already in Loki for 30 days, alongside the client IP. Capturing failed searches carefully in the application while Caddy logs every search verbatim would be theatre. The Caddy log needs the query string stripped (or the field removed) in the same pass.

5.5 Legal housekeeping

• Lawful basis: consent (Art. 6(1)(a)) for Tier B; Tier A processes no personal data at the sink; transient IP handling for rate limiting rests on legitimate interest (Art. 6(1)(f)) and is security-necessary.
• DPIA: still not triggered — no systematic large-scale monitoring, no profiling with legal effect, no special categories, and the scale is ~3k human requests/day. Worth a one-page note recording why it was not triggered.
• Art. 30 record of processing: needs two lines now, not one — the anonymous counts and the consented session statistics, with their separate retentions.
• Processors: none. Everything stays on hardware the operator controls.

---

6. Decisions still needed from the operator

Decisions 1–3 are settled and built (or, for IPs, documented). What follows is what is still open. The flags stay off until these are answered.

6.1 — Approve the privacy-page copy (§7). It is written and tested but it is a public legal statement in the operator's name. Nothing turns on until it is approved and shipped. Specifically confirm the numbers in it are the ones you intend to honour: 400 days (anonymous), 30 days (session), 90 days (failed searches), 30 min / 2 h (id rotation).

6.2 — Time-box Tier B, or run it permanently? Strong recommendation: time-box it. Turn it on for a defined study window (6–8 weeks), answer the specific funnel questions, roll up, then turn the flag off and keep Tier A running. The banner then disappears again. A permanent banner is a permanent tax on every first visit, for a question that is mostly asked once.

6.3 — Failed-search retention and the k-floor. Built with SEARCH_MISS_MIN_COUNT = 3 (strings fewer than 3 people typed are deleted after 7 days) and 90-day overall retention. Confirm, or tighten. Raising k to 5 costs little and is the cheapest way to buy margin on the only free-text field.

6.4 — Who may read Tier B? Per-event session rows are the most sensitive thing the project stores. Today anyone with database access can read them. Decide whether that is acceptable or whether a restricted role / view-only rollup access is wanted.

6.5 — Environment separation. Dev and beta run the same code. Recommended: Tier A on dev during validation then off; Tier B never on dev or beta (real consent from real visitors is not obtainable on a staging box, and dev traffic would pollute the funnels). Confirm.

6.6 — "Forget this session" endpoint? A one-click withdrawal could additionally send the live session id to a delete endpoint, so the current session's rows are erased rather than merely expiring in 30 days. Costs: a new public mutating endpoint that takes an id (unguessable at 128 bits, but still an endpoint), and a stronger deletion promise than can be kept for older sessions. Recommended: yes, as a follow-up, because "turn it off" that also erases what was already collected is a materially better promise. Not built.

6.7 — Consent re-ask intervals. Built as ~6 months for a grant, 12 months before ever re-asking a refusal. Confirm or change.

6.8 — Referrer beyond view.open. Unchanged from v1: kept only on view.open. Widening it would allow "which referral source converts" per session — genuinely useful now that Tier B exists, but it also makes a session more linkable to its arrival context. Recommended: leave as-is, and answer it by joining Tier A day-level referrer counts instead.

6.9 — Confirm GPC/DNT standing down Tier A as well. Built that way. It is stricter than required and costs a little coverage on the tier that needs no consent at all. Confirm, or restrict the GPC/DNT stand-down to Tier B only.

---

6b. Where I think these decisions are wrong

Asked for judgement, not compliance. Decisions 2 and 3 I agree with. Decision 1 I would push back on.

Decision 1 (session identifier) — I think this is a bad trade for this product, at this scale, right now. Four reasons, in order of weight:

1. The volume probably cannot support the analysis. ~3,000 human requests/day is on the order of 600–900 sessions/day. A plain, non-manipulative banner typically converts somewhere in the 20–60% band; take 35% and you have ~250 consented sessions/day. A four-step funnel on 250 sessions/day gives step rates with confidence intervals of several percentage points — so detecting anything short of a large effect takes weeks per comparison, and any A/B reading is out of reach entirely. The identifier buys a shape you could mostly have guessed, not a measurement you can act on.
2. The bias is correlated with the thing being measured (§5.3). Privacy- declining users and power users are plausibly the same people here.
3. The most valuable funnel needs no identifier at all. "Did the visitor who tapped Use my location reach a grade?" happens entirely within one page load on the homepage — a single JS context, no navigation, no storage. It is answerable today with an ordered pair of Tier A events and zero consent. Same for the picker→grade path. What genuinely needs the id is only the cross-view journey (home → calendar → compare), which is the less pressing question.
4. It costs the product's clearest differentiator. The footer says "No ads. No tracking." and the privacy page was, unusually, entirely true. Replacing that with a consent banner is a real product cost on a site whose appeal is partly that it does not behave like everything else.

What I would do instead: ship Tier A, run it for a month, and write down the specific questions it failed to answer. If any survive, turn Tier B on as a time-boxed study (§6.2) rather than as permanent instrumentation. The prototype is built so this costs nothing — Tier B is a separate flag and a separate table, and turning it on later requires no migration of Tier A data.

Decision 2 (failed-search text) — agreed, with one correction. The design mitigations (server-side only, reject-don't-sanitise, per-day counts, k≥3 prune) are, I think, sufficient. But the mitigations are worthless until the Caddy log is fixed (§5.4): Caddy currently logs the full request URI including ?q=<query> for every search, hit or miss, unnormalised, next to the client IP, for 30 days. The careful in-app capture is strictly less exposure than what is already happening by accident. Fix the Caddy log format in the same pass or decision 2 is a net decrease in privacy only on paper.

Decision 3 (drop/truncate IPs) — agreed, no reservations. The only cost is weaker retrospective abuse forensics, which is a fair price. Note the rate limiter must keep full precision in memory (§5.4).

---

7. Privacy-page copy

Written and shipped in this change: frontend/templates/privacy.html.j2. The substantive edits from the live page:

• "Analytics" → "Usage statistics", split into Anonymous counts — always on and Session statistics — only if you say yes, so the consent boundary is the organising structure of the section rather than a footnote.
• The claim "no per-visitor identifier" is removed — it becomes false.
• The session id is described concretely: session storage not a cookie, current tab only, 30 min idle / 2 h absolute rotation, gone when the tab closes, never linked to the account or to an IP, 30-day retention.
• The withdrawal path is stated, and so is its limit: turning it off deletes the id immediately and stops recording, but past records cannot be found and deleted because they are not linked to anyone — with the short retention given as the reason that is acceptable.
• GPC/DNT are named explicitly as being honoured.
• A new "Searches that found nothing" subsection: what is kept, what is discarded, the k-anonymity prune, and that successful searches are never stored as text.
• A new "Server logs" subsection stating that addresses are shortened before being written — this must not ship before the logging-pipeline change actually truncates them, or the page becomes untrue in the other direction.
• The location section now states positively that coordinates never enter the usage statistics — only how a place was picked, never where.
• The footer's blanket "No tracking." becomes "No third-party trackers." on both the base template and the homepage hero, which is the claim that stays true.

Tests assert the load-bearing promises (frontend/tests/unit/test_pages.py), so if the copy and the code drift apart the suite fails rather than the page quietly becoming a false statement.

---

8. Implementation plan

Backend (backend/)

File	Change	Status
core/events.py	SCHEMA, normalize(), hourly upsert, JSONL sink; v2: three flags, validate_session/validate_seq, session-row insert, normalize_search_miss + record_search_miss, PRUNE_SEARCH_MISS_SQL	done
core/metrics.py	rate_ok() so one event spends one token across sinks; per-IP ceiling 120/min, env-tunable; reset_rate_limiter() for tests	done
web/app.py	api_event: batching, streaming body cap, Sec-Fetch-Site check, v2: session/seq pass-through. api_suggest: place.search + v2: record_search_miss on zero results	done
alembic/…/0003_ui_events.py	Tier A ui_event_hourly hypertable, 400-day retention	done
alembic/…/0004_ui_event_sessions.py	v2: ui_event_session hypertable (30-day retention, 1-day chunks) + ui_search_miss	done
daily maintenance job	run events.PRUNE_SEARCH_MISS_SQL; drop ui_search_miss rows past 90 days	todo
30-day rollup job	ui_event_session → anonymous session-outcome rows before the retention drop	todo
accounts/api_accounts.py	alert funnel steps recorded server-side	todo
web/app.py / content.py	deadend prop=not_found on 404s	todo
tests/core/test_events.py	allowlist, slot dropping, no-free-text-by-construction, flag-off inertness, never-raises; v2: session-shape validation, seq bounds, tier independence, id-never-in-logs, search-miss normalisation + rejection + k-floor	done (21)
tests/web/test_event_beacon.py	204 uniformity, batch cap, oversize drop, cross-site drop; v2: session/seq over the wire, forged-session rejection, sessions-flag independence, auth-cookie never read	done (15)
tests/web/test_api.py	v2: miss text captured server-side only, hits never stored, never in the app access log	done (2)

Frontend (frontend/)

File	Change	Status
static/consent.js	v2, new — opt-in banner, equal-weight buttons, localStorage consent record with re-ask intervals, GPC/DNT stand-down, footer withdrawal control, onConsentChange	done
static/track.js	flag check, batching, currentView(), [data-event] delegation, trackLegacy(); v2: consent gate, session minting/rotation/caps, seq, immediate drop on withdrawal	done
static/style.css	v2: .consent-bar / .consent-control, tokens only, 44px targets, mobile stacking	done
static/digest.js	beacon moved out; re-exports for importers	done
static/app.js, mappicker.js, units.js	view.open, place.pick, deadend, share, view.control call sites	done
static/*.html (6 shells)	load consent.js then track.js	done
templates/base.html.j2	flag stamp on <html>; v2: consent script, footer withdrawal control, "No third-party trackers"	done
templates/home.html.j2	v2: hero trust line no longer claims "No tracking"	done
templates/privacy.html.j2	v2: full rewrite (§7)	done
content.py, app.py	EVENTS_ENABLED on both HTML paths	done
static/calendar.js, compare.js, day.js, score.js, subscriptions.js	their own call sites	todo
tests/unit/test_pages.py	flag off by default / stamped when on; v2: consent control present everywhere, script order, privacy-copy promises, footer claim	done (6)

Rollout

1. Land with all three flags unset everywhere. Inert: the beacon answers 204, no sink is written, no banner renders, no id is minted, and the flag stamp is absent from every page (asserted by test).
2. Ship the privacy-page copy and the IP truncation first — including the Caddy log-format fix (§5.4). The page must not claim addresses are shortened before they are.
3. Migrate + enable Tier A only on dev, then beta, then prod. Verify in Grafana (tag="ui.event") that no free text or coordinate ever appears.
4. Run Tier A for a month. Write down which questions it failed to answer (§6b).
5. Only then, and only if step 4 produced surviving questions: enable THERMOGRAPH_SEARCH_MISS and THERMOGRAPH_EVENT_SESSIONS on prod only, as a time-boxed study. Watch the consent accept rate and compare Tier B's rates against Tier A's to quantify the bias before trusting any funnel.
6. Roll up and turn Tier B off at the end of the window; the banner disappears with it.
7. Kill switch: unset a flag. No deploy, no migration, no data loss — and unsetting THERMOGRAPH_EVENT_SESSIONS alone removes the banner and the identifier while leaving the anonymous history intact.

Cost check

~3,000 human req/day → ~1–3k events/day, batched ≈ a few hundred extra requests/day (~0.005 req/s). Tier A grows a few hundred rows/day. Tier B, at a 35% consent rate and ~10 events/session, is on the order of 2–3k rows/day held for 30 days — under a million rows at steady state, trivially small. ui_search_miss is bounded by distinct failed queries per day, realistically tens.