Compare commits
2 commits
dev
...
design/ui-
| Author | SHA1 | Date | |
|---|---|---|---|
|
35b58300cb | ||
|
|
0ddc457d8c |
28 changed files with 2661 additions and 86 deletions
672
UI-EVENTS.md
Normal file
672
UI-EVENTS.md
Normal file
|
|
@ -0,0 +1,672 @@
|
|||
# UI product-event instrumentation — design (v2)
|
||||
|
||||
**Status: design + prototype. Every tier feature-flagged OFF. Not deployed
|
||||
anywhere.** Enabling any of it requires the decisions in
|
||||
[§6](#6-decisions-still-needed-from-the-operator) and the privacy-page copy in
|
||||
[§7](#7-privacy-page-copy) to ship first.
|
||||
|
||||
The goal is to learn what people actually *do* in the Thermograph UI — did they
|
||||
ever get a location, does search work, which views earn their maintenance.
|
||||
|
||||
**v2 splits the answer into two tiers**, because the operator's decisions
|
||||
changed what is being built:
|
||||
|
||||
* **Tier A — anonymous aggregates.** No identifier, nothing stored on the
|
||||
device, no consent needed, runs for every visitor. This was the whole of v1
|
||||
and is still the primary signal.
|
||||
* **Tier B — consented session rows.** An ephemeral per-tab identifier so
|
||||
funnels can be computed. Opt-in consent, short retention, its own flag.
|
||||
|
||||
Keeping them separate is the single most important structural choice in v2: if
|
||||
consent rates are poor (they usually are), Tier A still answers most of §1, and
|
||||
Tier B's numbers can be *calibrated against* Tier A to measure the consent bias
|
||||
rather than pretending it isn't there.
|
||||
|
||||
---
|
||||
|
||||
## What changed from v1, and why
|
||||
|
||||
| # | v1 | v2 | Why |
|
||||
|---|---|---|---|
|
||||
| 1 | **No identifier at all.** Counts of interactions, never of people. | **Ephemeral per-tab session id**, opt-in, 30 min idle / 2 h absolute rotation, sessionStorage. Tier A unchanged and still runs for everyone. | Operator decision 1: rates can't answer "did *this visitor* succeed". |
|
||||
| 2 | No consent banner needed (nothing stored on the device). | **Opt-in consent banner required** for Tier B, with equal-weight Allow/No, one-click withdrawal in the footer of every page, GPC/DNT treated as a refusal already given. | Storing an identifier engages ePrivacy Art. 5(3); analytics is not "strictly necessary". §5.1 argues the exemption case and rejects it. |
|
||||
| 3 | `ui_event_hourly` aggregate only — deliberately incapable of holding a behavioural sequence. | Adds `ui_event_session` (one row per event), **30-day retention**, minute-granularity timestamps, `seq` for ordering, 200 events/session cap. | An aggregate cannot express a funnel. This *is* the sequence dataset v1 avoided, so it gets the shortest retention in the system. |
|
||||
| 4 | Failed search text never captured (§6.2 deferred it). | **Captured server-side** in `api_suggest`: normalised, rejected on any personal-data smell, stored as a per-day count, k-anonymity prune at 3. | Operator decision 2. |
|
||||
| 5 | Raw IP in `audit.log_access` flagged as an open question. | Treated as **decided: truncate**. Not implemented here (logging-pipeline work owns it); §5.4 states what the app side must do so the two changes stay consistent. | Operator decision 3. |
|
||||
| 6 | Privacy-page rewrite listed as a follow-up. | **Written and included** (`frontend/templates/privacy.html.j2`), with tests asserting the load-bearing promises. | The v1 page becomes untrue the moment Tier B ships. |
|
||||
| 7 | One flag, `THERMOGRAPH_EVENTS`. | Three: `THERMOGRAPH_EVENTS` (Tier A), `THERMOGRAPH_EVENT_SESSIONS` (Tier B), `THERMOGRAPH_SEARCH_MISS` (free text). | Each is a separate decision with a separate risk profile and should be separately revocable. |
|
||||
|
||||
**Unchanged from v1:** the seven events and their allowlists, the beacon
|
||||
transport, the abuse-resistance layers, the server-side-derivation preference,
|
||||
and the rejected storage alternatives.
|
||||
|
||||
---
|
||||
|
||||
## 0. What already exists (build on this, don't reinvent)
|
||||
|
||||
| Seam | What it does | Where |
|
||||
|---|---|---|
|
||||
| Request middleware | classifies + counts every inbound request, writes an access JSONL line **including the raw client IP** | `backend/web/app.py` `revalidate_static` |
|
||||
| `core/metrics.py` | since-start counters (inbound by category, outbound by source, heartbeats) behind `GET /api/v2/metrics` | `backend/core/metrics.py` |
|
||||
| **Product-event beacon** | `POST /api/v2/event`, `navigator.sendBeacon`, allowlisted names, per-IP rate limit, referrer reduced to a bare domain, always 204 | `api_event` + `metrics.record_event` |
|
||||
| Event counters | `(event, referrer_domain, UTC day)` → count, **7-day** window, UNLOGGED, TRUNCATEd every deploy | `metrics.EVENTS`, `event_cume` |
|
||||
| Client tracker | `track(event)` + `[data-event]` click delegation | `frontend/static/digest.js` |
|
||||
| Activity JSONL | `audit.log_activity(tag, record)` → `logs/activity/*.jsonl` | `backend/core/audit.py` |
|
||||
| Loki/Grafana | Alloy ships `/applogs/**/*.jsonl` fleet-wide, **30-day** retention | `observability/` |
|
||||
|
||||
So four coarse events (`home.locate`, `home.digest_signup`, `home.records_click`,
|
||||
`home.share`, plus `home.nav_*`) are already live, on the SSR homepage only.
|
||||
|
||||
**Three gaps this design closes.**
|
||||
|
||||
1. **No dimensions.** `home.locate` is a bare counter. It cannot say whether the
|
||||
tap *succeeded*, which is the only interesting part.
|
||||
2. **No durability.** The counters are UNLOGGED and TRUNCATEd on every deploy,
|
||||
and capped at 7 days. Loki holds 30 days of JSONL. Neither survives a quarter,
|
||||
so no seasonal or before/after comparison is possible — which is most of the
|
||||
value for a site about *weather*.
|
||||
3. **Nothing outside the homepage.** Calendar, Day, Compare, Score, Alerts — the
|
||||
whole interactive product — is uninstrumented.
|
||||
|
||||
Two corrections to assumptions that were made when this work was scoped:
|
||||
Loki retention is **30 days** (`observability/loki/config.yml`, `720h`), not 24h;
|
||||
and the Alloy glob `/applogs/**/*.jsonl` already picks up the `activity/` stream,
|
||||
so a new JSONL sink needs no shipping config.
|
||||
|
||||
A third, found while designing v2 and material to decision 2: the `log` block in
|
||||
`infra/deploy/Caddyfile` sets no `format`, so Caddy's **default JSON logger**
|
||||
records `request.uri` — the full URI **including the query string**. Every
|
||||
`/api/v2/suggest?q=…` a visitor types is therefore already in Loki for 30 days,
|
||||
next to their IP. See §5.4.
|
||||
|
||||
---
|
||||
|
||||
## 1. The events worth capturing
|
||||
|
||||
Traffic is ~9,700 req/day of which ~60% is crawlers, so **~3,000 human req/day**.
|
||||
At that volume a firehose buys nothing a well-chosen aggregate doesn't, and every
|
||||
extra event is another thing to keep honest. Seven events, each tied to a decision
|
||||
someone would actually make:
|
||||
|
||||
| Event | Question it answers | Decision it informs |
|
||||
|---|---|---|
|
||||
| `view.open` | How many sessions reach each view, and from where? | The denominator for everything else; whether Compare/Score justify their maintenance |
|
||||
| `place.pick` | Does anyone ever get past "no place picked", and by which route (geolocate / map / search / restore / link)? | **The activation question.** If geolocate is mostly declined, promote the map picker |
|
||||
| `place.search` | Does location search find anything? (`hit` / `typo` / `miss`) | Whether to invest in the geocoder/typo tolerance |
|
||||
| `view.control` | Which in-view controls get used — chart metric, °F/°C, date, range, season, compare add/remove, distribution metric | Which chart metrics to keep/promote; whether the unit default is wrong |
|
||||
| `share` | Is the permalink / PNG export used at all? | Whether to keep maintaining the SVG→PNG exporter |
|
||||
| `alert` | The alerts funnel: open → start → create, push on/off | Where subscription signup leaks |
|
||||
| `deadend` | Errors the visitor actually *saw*: grade error, no data, geolocation denied, search miss, 404, offline | Bug reports nobody filed |
|
||||
|
||||
**Deliberately not captured** — because the request log already answers them, and
|
||||
duplicating a signal means two numbers that will eventually disagree: page-view
|
||||
counts, API latency/error rates, and asset traffic. Also deliberately not
|
||||
captured: hover/scroll/dwell (individuating, and answers nothing on this list),
|
||||
map pan/zoom (very high volume, low decision value, and pan/zoom coordinates
|
||||
are location data), and anything at all about *which* place was chosen.
|
||||
|
||||
**v2 note:** the event list is unchanged. Tier B adds a session id to these same
|
||||
seven events; it does not add an eighth. That is deliberate — the consent
|
||||
conversation is about *linking* what was already being counted, not about
|
||||
collecting more, and it keeps the privacy-page copy describing one list.
|
||||
|
||||
### Two events are recorded server-side, not in the browser
|
||||
|
||||
`place.search` is recorded in `api_suggest`: the server already knows the answer,
|
||||
every surface that searches is covered for free, and **the query string never has
|
||||
to leave the request handler** — only the three-way outcome is stored, plus (v2,
|
||||
decision 2) the normalised text of the *misses* only. Same
|
||||
argument would apply to alert creation (`accounts/api_accounts.py`). Prefer this
|
||||
whenever the server can derive the fact: less client code, no forgeability, and
|
||||
strictly less data in flight.
|
||||
|
||||
---
|
||||
|
||||
## 2. Schema
|
||||
|
||||
Small, typed, and *closed*. Three dimension slots (`view`, `prop`, `value`) plus
|
||||
a referrer domain on `view.open` only. Every value is an enum from a fixed
|
||||
allowlist (`backend/core/events.py` `SCHEMA`). Unknown event name → one `other`
|
||||
bucket with blank dimensions; unknown value in a declared slot → `other`; a slot
|
||||
an event doesn't declare → dropped entirely. **Nothing a client sends is ever
|
||||
persisted verbatim.**
|
||||
|
||||
```
|
||||
view.open view=<VIEWS> prop=direct|nav|link|restore +referrer domain
|
||||
place.pick view=<VIEWS> prop=geolocate|map|search|restore|link
|
||||
place.search prop=hit|typo|miss
|
||||
view.control view=<VIEWS> prop=<CONTROLS> value=<VALUES>
|
||||
share view=<VIEWS> prop=link|png|permalink
|
||||
alert prop=open|start|create|push_on|push_off|test value=y|n
|
||||
deadend view=<VIEWS> prop=grade_error|no_data|geo_denied|search_miss|not_found|offline
|
||||
```
|
||||
|
||||
`VIEWS` = home, calendar, day, compare, score, alerts, legend, city, cities,
|
||||
records, month, glossary, about, privacy.
|
||||
`CONTROLS` = chart_metric, unit, date, range, season, totals, dist_metric,
|
||||
comfort, compare_add, compare_remove, filter_sheet, day_step.
|
||||
`VALUES` = the metric keys, `F`/`C`, `today`/`past`/`future`, count buckets
|
||||
(`1`…`5plus`), span buckets (`1y`…`max`), season keys, `on`/`off`, `other`.
|
||||
|
||||
Numbers are bucketed **client-side** before they are sent (`5plus`, not `7`):
|
||||
a raw count is a small amount of entropy, and a bucket is all the decision needs.
|
||||
|
||||
**Tier A storage row:** `(hour_bucket, event, view, prop, value, referrer) →
|
||||
count`. No row per interaction, no column an identifier could go in. Worst-case
|
||||
key space is a few hundred rows/hour; realistically ~50.
|
||||
|
||||
**Tier B storage row:** `(minute_ts, session, seq, event, view, prop, value,
|
||||
referrer)` — one row per event, written *only* when the visitor consented. The
|
||||
schema above is unchanged; the session id and sequence number are the only
|
||||
additions, and no new dimension is collected because of Tier B.
|
||||
|
||||
### The Tier B identifier, specified exactly
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Value** | 16 bytes from `crypto.getRandomValues`, base64url, 22 chars. Server-validated by exact shape (`^[A-Za-z0-9_-]{22}$`) — see below. |
|
||||
| **Where it lives** | `sessionStorage`, key `thermograph:sid`. Not `localStorage` (would make it a stable pseudonym across visits — the exact thing being avoided). Not a cookie (would ride on every request including static assets, be readable on routes unrelated to analytics, and buy nothing). |
|
||||
| **Lifetime** | Rotates after **30 min idle**; rotates after **2 h absolute** regardless of activity; dies with the tab. |
|
||||
| **Scope** | One tab. Two tabs = two sessions; a return visit = a new session; a second device = unrelated. **No cross-session, cross-tab, or cross-device linkage exists to break** — there is nothing stored that could join them. |
|
||||
| **Cap** | 200 events per session, then Tier B stops and Tier A carries on. Bounds both storage and how detailed any one sequence can get. |
|
||||
| **Minted** | Only inside `hasConsent()`. Refusing, ignoring the banner, or signalling GPC/DNT writes *nothing at all* to the device. |
|
||||
| **Account linkage** | **Never.** Argued below. |
|
||||
| **Retention** | 30 days, then the chunk is dropped. |
|
||||
|
||||
**Why the session id is never joined to the account id** (the decision said
|
||||
"unless you argue otherwise" — the argument is for *never*):
|
||||
|
||||
1. Joining converts an anonymous behavioural sequence into personal data about an
|
||||
*identified* person, which attaches subject-access, rectification and erasure
|
||||
duties to the entire event history and makes the 30-day retention a floor
|
||||
rather than a nicety.
|
||||
2. The analytical gain is close to zero: every question in §1 is about the
|
||||
*product* ("does the picker convert?"), not about *who*.
|
||||
3. Signed-in visitors are a small minority, so an account-linked subset is a
|
||||
biased sample you could not generalise from anyway — it would be the *most*
|
||||
engaged users, i.e. exactly the ones whose funnel is least representative.
|
||||
4. It is enforced structurally, not by policy: `api_event` never reads the auth
|
||||
cookie, `sendBeacon` cannot attach credentials cross-origin, and there is no
|
||||
user column in `ui_event_session`. There is a test for it.
|
||||
|
||||
**The session field is the only client-controlled string that reaches storage**,
|
||||
so it is validated by shape and dropped *whole* on any mismatch rather than
|
||||
sanitised into shape. A trimmed-to-fit id would still be attacker-chosen text; a
|
||||
rejected one is nothing. Tier A records identically either way, so refusing a
|
||||
malformed id costs the primary signal zero.
|
||||
|
||||
Migration note: the four live `home.*` events keep their current unflagged,
|
||||
un-dimensioned behaviour so the ops endpoint's numbers don't reset when this
|
||||
lands. Fold them into `view.open`/`place.pick`/`share` and retire the old names
|
||||
as a follow-up, once there's overlap data to stitch the series.
|
||||
|
||||
---
|
||||
|
||||
## 3. Transport and storage
|
||||
|
||||
**Chosen: the existing beacon → two sinks.**
|
||||
|
||||
* **Transport** — extend `POST /api/v2/event` (`navigator.sendBeacon`, batched,
|
||||
always 204). Already built, already excluded from the traffic counters, already
|
||||
rate-limited. Batching (≤20 events, flushed on a 4 s timer and unconditionally
|
||||
on `pagehide`/`visibilitychange`) matters because chart-metric taps and slider
|
||||
drags arrive in bursts.
|
||||
* **Durable sink** — `ui_event_hourly`, a TimescaleDB hypertable
|
||||
(`backend/alembic/versions/0003_ui_events.py`): 30-day chunks, a native
|
||||
retention policy, and an `ON CONFLICT … count + 1` upsert. The DB is already
|
||||
TimescaleDB with hypertables and a compression/retention pattern to copy.
|
||||
* **Live sink** — one `audit.log_activity("ui.event", …)` JSONL line per event.
|
||||
Alloy already ships that path, so the last 30 days are queryable in Grafana
|
||||
immediately without waiting on the rollup. Same allowlisted enums, nothing else.
|
||||
* **Tier B sink** — `ui_event_session`, one row per event, 1-day chunks, **30-day**
|
||||
retention policy. Timestamps truncated to the **minute** and ordering carried by
|
||||
an in-session `seq` instead: precise clock times are a fingerprint (millisecond
|
||||
inter-event gaps individuate, and adjacency lets two rotated sessions be
|
||||
stitched back together), and a funnel needs order, not timing. No IP column, no
|
||||
user-agent column, no account column, and no unique constraint that would
|
||||
require one.
|
||||
* **Rollup** — at 30 days, per-event rows are aggregated into anonymous
|
||||
session-outcome rows (`day, entry_view, reached_pick, reached_grade, n`) with no
|
||||
identifier, retained alongside Tier A. The sequence data expires; the learning
|
||||
does not. *(Specified here, not yet implemented — see §7.)*
|
||||
* **The existing in-process counters stay untouched** — they remain the live
|
||||
"what's happening right now" view on `/api/v2/metrics`.
|
||||
|
||||
**Why not the alternatives:**
|
||||
|
||||
* *Loki only.* 30-day retention loses every seasonal comparison — the thing a
|
||||
weather product most wants. Loki is also a poor aggregation engine for
|
||||
"unit switches per week over a year". Kept as the live tier, rejected as the
|
||||
system of record.
|
||||
* *Extend the in-process counters.* Cheapest option and no new storage, but they
|
||||
are UNLOGGED, TRUNCATEd every deploy, capped at 7 days, and have exactly one
|
||||
dimension. Every question in §1 needs at least two. Kept, not extended.
|
||||
* *Per-event rows for **everyone** (a real firehose).* Still rejected. v2 adds
|
||||
per-event rows, but only for the consented minority and only for 30 days.
|
||||
Making them the default would put every visitor's behavioural sequence in the
|
||||
database to answer questions that Tier A's aggregate already answers, and
|
||||
would drag the whole dataset behind the consent banner.
|
||||
* *Session-aggregate instead of per-event rows for Tier B* (one row per session:
|
||||
`entry_view, reached_pick, reached_grade, n_controls`). Genuinely tempting —
|
||||
it is strictly less data and answers the headline funnel. Rejected for the
|
||||
**live** table because the client would have to decide when a session is
|
||||
"done" (it never reliably knows — tabs are closed, not ended) and because a
|
||||
fixed set of outcome columns forecloses any funnel question not thought of in
|
||||
advance. It *is* adopted as the **rollup**: at 30 days the per-event rows are
|
||||
aggregated into exactly this shape, with no session id, and kept 400 days —
|
||||
so the sequence data is short-lived and the learning is permanent.
|
||||
* *A third-party analytics SaaS (Plausible/Umami/GA).* GA is a non-starter
|
||||
(US transfer, consent banner, cookies). Self-hosted Plausible/Umami would be
|
||||
defensible, but it means another service to run and its own identifier model
|
||||
(Plausible hashes IP+UA+salt — a pseudonymous identifier, which is precisely
|
||||
the decision being deferred to the operator in §6.1). Rejected as more moving
|
||||
parts for less control.
|
||||
|
||||
---
|
||||
|
||||
## 4. Abuse and bot resistance
|
||||
|
||||
The endpoint is public and unauthenticated on a site where 60% of traffic is
|
||||
crawlers and scanners. Layered, in order of how much they actually buy:
|
||||
|
||||
1. **Closed allowlist** (the real defence). Nothing outside `SCHEMA` is stored;
|
||||
unknown names collapse into one `other` bucket. Storage is bounded by the
|
||||
*product of the enums*, not by what anyone sends. The endpoint cannot be grown
|
||||
into a spam sink or a data-exfil channel.
|
||||
2. **Body cap, enforced while streaming** — `MAX_BODY_BYTES = 4096`; a declared
|
||||
`Content-Length` past it is refused without reading, and an undeclared one
|
||||
stops mid-stream. (The cap must comfortably exceed a full 20-event batch, or
|
||||
the byte cap silently eats legitimate traffic instead of abusive traffic —
|
||||
there is a regression test for exactly that.)
|
||||
3. **Per-IP rate limit** — the existing token bucket, raised from 30/min to
|
||||
**120/min** (`THERMOGRAPH_EVENT_RATE_PER_MIN`) because 30 was sized for four
|
||||
coarse events and a batched interaction stream would silently truncate normal
|
||||
use, biased toward the most engaged visitors. The IP is a key in a map cleared
|
||||
every minute; it is never written anywhere.
|
||||
4. **Soft first-party check** — `Sec-Fetch-Site` must be `same-origin`/
|
||||
`same-site`/`none`; a `cross-site` value is dropped, and a POST with neither
|
||||
`Sec-Fetch-Site` nor `Origin` is flagged as the first bucket to look at.
|
||||
Every browser that can run our JS sets this; naive scanners don't. Explicitly
|
||||
**not** a security boundary — anything that bothers can spoof it.
|
||||
5. **Uniform 204** — counted, rejected, over-cap, and rate-limited are
|
||||
indistinguishable, so probing tells an attacker nothing.
|
||||
6. **No UA-based bot filtering.** Blocking on User-Agent is a coin flip and gives
|
||||
false confidence. The structural defence is that the beacon requires JS
|
||||
execution and is linked from nowhere.
|
||||
|
||||
7. **Session-field shape validation** (new in v2). `session` is the one
|
||||
client-controlled string that reaches storage. It is matched against
|
||||
`^[A-Za-z0-9_-]{22}$` and dropped whole on mismatch — never trimmed into
|
||||
shape, because trimmed attacker text is still attacker text. A forged id also
|
||||
cannot inflate anything: Tier A is keyed on enums, not on sessions.
|
||||
8. **Unbounded session minting** is the one new abuse surface — an attacker can
|
||||
generate as many valid-looking ids as they like and manufacture fake
|
||||
"sessions". The per-IP rate limit bounds it to 120 rows/min/IP, the 200-event
|
||||
cap bounds any single session, and the 30-day retention bounds the blast
|
||||
radius. The detection signal is the same cross-check as below: Tier B's
|
||||
session count should stay in a stable ratio to Tier A's `view.open` count, and
|
||||
a sudden divergence means manufactured sessions.
|
||||
|
||||
**Poisoning** (someone inflating a number rather than exhausting storage) cannot
|
||||
be fully prevented without identity. Mitigation is cross-checking: the server's
|
||||
own inbound page counts are an independent measure of `view.open`, so a
|
||||
divergence between them is the alarm. Any analysis should treat these numbers as
|
||||
*directional*, never as revenue-grade.
|
||||
|
||||
---
|
||||
|
||||
## 5. Privacy position
|
||||
|
||||
The site is EU-hosted with EU visitors, so GDPR + ePrivacy apply. The public
|
||||
privacy page promised, verbatim: *"no analytics library, no cookies for tracking,
|
||||
and no per-visitor identifier… simple aggregate counts."* **Tier B makes that
|
||||
false**, which is why the rewritten copy (§7) is part of this change and not a
|
||||
follow-up.
|
||||
|
||||
### 5.1 Consent — the argument, and why I land on "banner"
|
||||
|
||||
The question is whether Tier B can run without opt-in. It cannot, and here is the
|
||||
reasoning rather than an assumption.
|
||||
|
||||
**ePrivacy Art. 5(3) is the binding constraint, not GDPR.** It covers *any*
|
||||
storing of, or access to, information on terminal equipment — not just cookies,
|
||||
so `sessionStorage` is squarely in scope. It admits exactly two exemptions:
|
||||
transmission-necessary, and strictly necessary for a service *explicitly
|
||||
requested by the user*. Nobody visits a weather site to be measured, so the
|
||||
second exemption does not fit on its plain reading. Crucially, **legitimate
|
||||
interest is not available here at all** — it is a GDPR Art. 6 lawful basis, and
|
||||
it cannot substitute for the Art. 5(3) consent requirement. Any argument of the
|
||||
form "we have a legitimate interest in analytics, so no banner" is simply
|
||||
answering the wrong question.
|
||||
|
||||
**The one genuinely arguable route** is the audience-measurement exemption:
|
||||
CNIL's guidance (and similar positions from a few other DPAs) exempts a
|
||||
first-party measurement identifier where it is used *solely* for aggregate
|
||||
statistics, is not shared, is not cross-site, is not combined with other
|
||||
processing, and has bounded lifetimes. This design would fit that description
|
||||
comfortably — arguably better than the tools the exemption was written for, since
|
||||
the id here dies in 2 hours rather than 13 months. So the argument is real, and
|
||||
I am not dismissing it.
|
||||
|
||||
**Why I still recommend the banner:**
|
||||
|
||||
1. It is a **national regulator's exemption, not harmonised EU law**. Germany's
|
||||
TDDDG §25 is read more narrowly, and the EDPB has never blessed a general
|
||||
analytics exemption. EU visitors are not all French.
|
||||
2. The exemption is conditioned on use — "solely for aggregate statistics". Tier
|
||||
B exists precisely to look at *sequences* within a session. That is still
|
||||
aggregate at the reporting layer, but it is a harder story to tell than a
|
||||
pageview counter, and the conditions are assessed on what you actually do.
|
||||
3. The downside is asymmetric. A banner costs a UX annoyance and some sample
|
||||
bias. Getting the exemption wrong on a site with no legal budget costs
|
||||
correspondence with a regulator.
|
||||
4. **Tier A makes the banner cheap.** Because the anonymous tier runs regardless,
|
||||
a refusal costs the *funnels*, not the *numbers*. If Tier A were also behind
|
||||
the banner I would push much harder on the exemption.
|
||||
|
||||
**The consent mechanism** (`frontend/static/consent.js`), designed to be a real
|
||||
choice rather than a compliance prop:
|
||||
|
||||
* Nothing is stored and **no id is minted before an explicit "Allow"** — ignoring
|
||||
the banner leaves exactly as little behind as refusing it. There is no
|
||||
"pending" state that quietly collects.
|
||||
* "Allow" and "No thanks" are the **same size, weight and prominence**. No
|
||||
pre-ticked boxes, no second-layer "legitimate interest" tab, no cookie-wall.
|
||||
* It is a **bar, not a modal**: it never blocks reading the page, which is the
|
||||
product.
|
||||
* **Withdrawal is as easy as granting** — an `Analytics: on/off` control in the
|
||||
footer of every page and on the privacy page. Withdrawing deletes the live
|
||||
session id immediately, discards anything queued-but-unsent, and stops further
|
||||
recording within the same page load.
|
||||
* **Re-asking:** a grant is re-confirmed after ~6 months; a refusal is **not**
|
||||
re-asked for 12 months. A banner that reappears after "no" is a dark pattern,
|
||||
not a question.
|
||||
* **GPC / DNT are treated as a refusal already given**: no banner is shown, no id
|
||||
is minted, and Tier A also stands down. Stricter than the law requires (GPC has
|
||||
no binding EU status), and it means the most privacy-conscious visitors are
|
||||
never nagged.
|
||||
* The **consent record itself** lives in `localStorage`. Storing a user's own
|
||||
choice is strictly necessary to honour it, so it is exempt from the consent it
|
||||
records — and it is deliberately the only thing that persists across sessions.
|
||||
|
||||
### 5.2 What is collected, tier by tier
|
||||
|
||||
| | Tier A (anonymous) | Tier B (consented) |
|
||||
|---|---|---|
|
||||
| Consent | not required | **opt-in required** |
|
||||
| Identifier | none | ephemeral, per-tab, 30 min idle / 2 h absolute |
|
||||
| On-device storage | none | one `sessionStorage` key |
|
||||
| Stored per interaction | nothing (hourly counts) | one row |
|
||||
| IP | never | never |
|
||||
| Account linkage | never | never |
|
||||
| Free text | never | never |
|
||||
| Coordinates | never | never |
|
||||
| Retention | 400 days | **30 days**, then anonymised rollup |
|
||||
| Runs for | everyone | consenting visitors only |
|
||||
|
||||
### 5.3 Consequences to be honest about
|
||||
|
||||
* **The consented sample is self-selected, and the bias is not random.** People
|
||||
who decline analytics skew technical and privacy-conscious — on a site about
|
||||
climate percentiles, that is plausibly *the same population* as the power users
|
||||
whose funnels are most interesting. Tier B may systematically under-represent
|
||||
exactly the behaviour it was added to study.
|
||||
* **Mitigation:** every Tier B rate has a Tier A counterpart computed over
|
||||
everyone. Comparing them measures the consent bias directly. If
|
||||
`place.pick / view.open` differs materially between the two, Tier B's funnels
|
||||
should be read as a *lower bound on a subpopulation*, not as the truth.
|
||||
* **We cannot honour an erasure request for past events**, because the whole
|
||||
point is that nothing links a session to a person — there is no lookup key.
|
||||
The mitigation is the 30-day retention, and the privacy page says so plainly
|
||||
rather than implying a deletion path that does not exist. *(A "forget this
|
||||
session" beacon carrying the live id would make it possible for the current
|
||||
session only — specified in §6.6, not built.)*
|
||||
* **Two tabs are two sessions**, so funnels **undercount** completion. That is
|
||||
the error direction to prefer, but it must not be forgotten when reading a
|
||||
number.
|
||||
|
||||
### 5.4 IP addresses (decision 3 — not implemented here)
|
||||
|
||||
The logging-pipeline work owns this. Recorded here so the two changes stay
|
||||
consistent:
|
||||
|
||||
* **The app side must stop writing raw IPs.** `audit.log_access` currently writes
|
||||
`_client_ip(request)` verbatim into `logs/access/*.jsonl`, which Alloy ships to
|
||||
Loki with 30-day retention. It should write a truncated form — **/24 for IPv4,
|
||||
/48 for IPv6** — or drop the field. Truncation preserves the geo-scale and
|
||||
abuse-pattern uses while stopping single-device identification.
|
||||
* **The rate limiter must keep the full IP in memory.** `metrics.rate_ok` needs
|
||||
per-address granularity; truncating its key to /24 would let one abuser
|
||||
rate-limit an entire NAT'd office. Full precision in a map cleared every
|
||||
minute, never written to disk, is the right split.
|
||||
* **Nothing in this design assumes durable raw IPs.** No event carries one, and
|
||||
no analysis depends on one. The only cost of truncation is weaker retrospective
|
||||
forensics if the beacon is ever abused — accepted.
|
||||
* **Caddy is the bigger leak, and it is in scope for decision 2.** The `log`
|
||||
block in `infra/deploy/Caddyfile` sets no `format`, so it uses Caddy's default
|
||||
JSON logger — which records `request.uri`, i.e. **the full URI including the
|
||||
query string**. Every `/api/v2/suggest?q=<what someone typed>` is therefore
|
||||
already in Loki for 30 days, alongside the client IP. Capturing failed searches
|
||||
carefully in the application while Caddy logs *every* search verbatim would be
|
||||
theatre. The Caddy log needs the query string stripped (or the field removed)
|
||||
in the same pass.
|
||||
|
||||
### 5.5 Legal housekeeping
|
||||
|
||||
* **Lawful basis:** consent (Art. 6(1)(a)) for Tier B; Tier A processes no
|
||||
personal data at the sink; transient IP handling for rate limiting rests on
|
||||
legitimate interest (Art. 6(1)(f)) and is security-necessary.
|
||||
* **DPIA:** still not triggered — no systematic large-scale monitoring, no
|
||||
profiling with legal effect, no special categories, and the scale is ~3k human
|
||||
requests/day. Worth a one-page note recording *why* it was not triggered.
|
||||
* **Art. 30 record of processing:** needs two lines now, not one — the anonymous
|
||||
counts and the consented session statistics, with their separate retentions.
|
||||
* **Processors:** none. Everything stays on hardware the operator controls.
|
||||
|
||||
---
|
||||
|
||||
## 6. Decisions still needed from the operator
|
||||
|
||||
Decisions 1–3 are settled and built (or, for IPs, documented). What follows is
|
||||
what is *still* open. The flags stay off until these are answered.
|
||||
|
||||
**6.1 — Approve the privacy-page copy (§7).** It is written and tested but it is
|
||||
a public legal statement in the operator's name. Nothing turns on until it is
|
||||
approved and shipped. Specifically confirm the numbers in it are the ones you
|
||||
intend to honour: 400 days (anonymous), 30 days (session), 90 days (failed
|
||||
searches), 30 min / 2 h (id rotation).
|
||||
|
||||
**6.2 — Time-box Tier B, or run it permanently?** Strong recommendation:
|
||||
**time-box it.** Turn it on for a defined study window (6–8 weeks), answer the
|
||||
specific funnel questions, roll up, then turn the flag off and keep Tier A
|
||||
running. The banner then disappears again. A permanent banner is a permanent tax
|
||||
on every first visit, for a question that is mostly asked once.
|
||||
|
||||
**6.3 — Failed-search retention and the k-floor.** Built with
|
||||
`SEARCH_MISS_MIN_COUNT = 3` (strings fewer than 3 people typed are deleted after
|
||||
7 days) and 90-day overall retention. Confirm, or tighten. Raising k to 5 costs
|
||||
little and is the cheapest way to buy margin on the only free-text field.
|
||||
|
||||
**6.4 — Who may read Tier B?** Per-event session rows are the most sensitive
|
||||
thing the project stores. Today anyone with database access can read them.
|
||||
Decide whether that is acceptable or whether a restricted role / view-only
|
||||
rollup access is wanted.
|
||||
|
||||
**6.5 — Environment separation.** Dev and beta run the same code. Recommended:
|
||||
Tier A on dev during validation then off; Tier B **never** on dev or beta (real
|
||||
consent from real visitors is not obtainable on a staging box, and dev traffic
|
||||
would pollute the funnels). Confirm.
|
||||
|
||||
**6.6 — "Forget this session" endpoint?** A one-click withdrawal could
|
||||
additionally send the live session id to a delete endpoint, so the current
|
||||
session's rows are erased rather than merely expiring in 30 days. Costs: a new
|
||||
public mutating endpoint that takes an id (unguessable at 128 bits, but still an
|
||||
endpoint), and a *stronger* deletion promise than can be kept for older sessions.
|
||||
Recommended: **yes, as a follow-up**, because "turn it off" that also erases what
|
||||
was already collected is a materially better promise. Not built.
|
||||
|
||||
**6.7 — Consent re-ask intervals.** Built as ~6 months for a grant, 12 months
|
||||
before ever re-asking a refusal. Confirm or change.
|
||||
|
||||
**6.8 — Referrer beyond `view.open`.** Unchanged from v1: kept only on
|
||||
`view.open`. Widening it would allow "which referral source converts" per session
|
||||
— genuinely useful now that Tier B exists, but it also makes a session more
|
||||
linkable to its arrival context. Recommended: **leave as-is**, and answer it by
|
||||
joining Tier A day-level referrer counts instead.
|
||||
|
||||
**6.9 — Confirm GPC/DNT standing down Tier A as well.** Built that way. It is
|
||||
stricter than required and costs a little coverage on the tier that needs no
|
||||
consent at all. Confirm, or restrict the GPC/DNT stand-down to Tier B only.
|
||||
|
||||
---
|
||||
|
||||
## 6b. Where I think these decisions are wrong
|
||||
|
||||
Asked for judgement, not compliance. Decisions 2 and 3 I agree with. Decision 1
|
||||
I would push back on.
|
||||
|
||||
**Decision 1 (session identifier) — I think this is a bad trade for this
|
||||
product, at this scale, right now.** Four reasons, in order of weight:
|
||||
|
||||
1. **The volume probably cannot support the analysis.** ~3,000 human requests/day
|
||||
is on the order of 600–900 sessions/day. A plain, non-manipulative banner
|
||||
typically converts somewhere in the 20–60% band; take 35% and you have
|
||||
~250 consented sessions/day. A four-step funnel on 250 sessions/day gives step
|
||||
rates with confidence intervals of several percentage points — so detecting
|
||||
anything short of a large effect takes weeks per comparison, and any A/B
|
||||
reading is out of reach entirely. The identifier buys a *shape* you could
|
||||
mostly have guessed, not a measurement you can act on.
|
||||
2. **The bias is correlated with the thing being measured** (§5.3). Privacy-
|
||||
declining users and power users are plausibly the same people here.
|
||||
3. **The most valuable funnel needs no identifier at all.** "Did the visitor who
|
||||
tapped *Use my location* reach a grade?" happens entirely within one page
|
||||
load on the homepage — a single JS context, no navigation, no storage. It is
|
||||
answerable today with an ordered pair of Tier A events and zero consent. Same
|
||||
for the picker→grade path. What genuinely needs the id is only the
|
||||
*cross-view* journey (home → calendar → compare), which is the less pressing
|
||||
question.
|
||||
4. **It costs the product's clearest differentiator.** The footer says "No ads.
|
||||
No tracking." and the privacy page was, unusually, entirely true. Replacing
|
||||
that with a consent banner is a real product cost on a site whose appeal is
|
||||
partly that it does not behave like everything else.
|
||||
|
||||
**What I would do instead:** ship Tier A, run it for a month, and write down the
|
||||
specific questions it failed to answer. If any survive, turn Tier B on as a
|
||||
time-boxed study (§6.2) rather than as permanent instrumentation. The prototype
|
||||
is built so this costs nothing — Tier B is a separate flag and a separate table,
|
||||
and turning it on later requires no migration of Tier A data.
|
||||
|
||||
**Decision 2 (failed-search text) — agreed, with one correction.** The design
|
||||
mitigations (server-side only, reject-don't-sanitise, per-day counts, k≥3 prune)
|
||||
are, I think, sufficient. But **the mitigations are worthless until the Caddy log
|
||||
is fixed** (§5.4): Caddy currently logs the full request URI including
|
||||
`?q=<query>` for *every* search, hit or miss, unnormalised, next to the client
|
||||
IP, for 30 days. The careful in-app capture is strictly less exposure than what
|
||||
is already happening by accident. Fix the Caddy log format in the same pass or
|
||||
decision 2 is a net *decrease* in privacy only on paper.
|
||||
|
||||
**Decision 3 (drop/truncate IPs) — agreed, no reservations.** The only cost is
|
||||
weaker retrospective abuse forensics, which is a fair price. Note the rate
|
||||
limiter must keep full precision in memory (§5.4).
|
||||
|
||||
---
|
||||
|
||||
## 7. Privacy-page copy
|
||||
|
||||
Written and shipped in this change: `frontend/templates/privacy.html.j2`. The
|
||||
substantive edits from the live page:
|
||||
|
||||
* **"Analytics" → "Usage statistics"**, split into *Anonymous counts — always on*
|
||||
and *Session statistics — only if you say yes*, so the consent boundary is the
|
||||
organising structure of the section rather than a footnote.
|
||||
* The claim **"no per-visitor identifier"** is removed — it becomes false.
|
||||
* The session id is described concretely: session storage not a cookie, current
|
||||
tab only, 30 min idle / 2 h absolute rotation, gone when the tab closes, never
|
||||
linked to the account or to an IP, 30-day retention.
|
||||
* **The withdrawal path is stated**, and so is its limit: turning it off deletes
|
||||
the id immediately and stops recording, but past records cannot be found and
|
||||
deleted *because* they are not linked to anyone — with the short retention
|
||||
given as the reason that is acceptable.
|
||||
* GPC/DNT are named explicitly as being honoured.
|
||||
* A new **"Searches that found nothing"** subsection: what is kept, what is
|
||||
discarded, the k-anonymity prune, and that successful searches are never stored
|
||||
as text.
|
||||
* A new **"Server logs"** subsection stating that addresses are shortened before
|
||||
being written — this must not ship before the logging-pipeline change actually
|
||||
truncates them, or the page becomes untrue in the other direction.
|
||||
* The location section now states positively that **coordinates never enter the
|
||||
usage statistics** — only *how* a place was picked, never where.
|
||||
* The footer's blanket **"No tracking."** becomes **"No third-party trackers."**
|
||||
on both the base template and the homepage hero, which is the claim that stays
|
||||
true.
|
||||
|
||||
Tests assert the load-bearing promises (`frontend/tests/unit/test_pages.py`), so
|
||||
if the copy and the code drift apart the suite fails rather than the page quietly
|
||||
becoming a false statement.
|
||||
|
||||
---
|
||||
|
||||
## 8. Implementation plan
|
||||
|
||||
### Backend (`backend/`)
|
||||
|
||||
| File | Change | Status |
|
||||
|---|---|---|
|
||||
| `core/events.py` | `SCHEMA`, `normalize()`, hourly upsert, JSONL sink; **v2:** three flags, `validate_session`/`validate_seq`, session-row insert, `normalize_search_miss` + `record_search_miss`, `PRUNE_SEARCH_MISS_SQL` | done |
|
||||
| `core/metrics.py` | `rate_ok()` so one event spends one token across sinks; per-IP ceiling 120/min, env-tunable; `reset_rate_limiter()` for tests | done |
|
||||
| `web/app.py` | `api_event`: batching, streaming body cap, `Sec-Fetch-Site` check, **v2:** `session`/`seq` pass-through. `api_suggest`: `place.search` + **v2:** `record_search_miss` on zero results | done |
|
||||
| `alembic/…/0003_ui_events.py` | Tier A `ui_event_hourly` hypertable, 400-day retention | done |
|
||||
| `alembic/…/0004_ui_event_sessions.py` | **v2:** `ui_event_session` hypertable (30-day retention, 1-day chunks) + `ui_search_miss` | done |
|
||||
| daily maintenance job | run `events.PRUNE_SEARCH_MISS_SQL`; drop `ui_search_miss` rows past 90 days | **todo** |
|
||||
| 30-day rollup job | `ui_event_session` → anonymous session-outcome rows before the retention drop | **todo** |
|
||||
| `accounts/api_accounts.py` | `alert` funnel steps recorded server-side | **todo** |
|
||||
| `web/app.py` / `content.py` | `deadend prop=not_found` on 404s | **todo** |
|
||||
| `tests/core/test_events.py` | allowlist, slot dropping, no-free-text-by-construction, flag-off inertness, never-raises; **v2:** session-shape validation, seq bounds, tier independence, id-never-in-logs, search-miss normalisation + rejection + k-floor | done (21) |
|
||||
| `tests/web/test_event_beacon.py` | 204 uniformity, batch cap, oversize drop, cross-site drop; **v2:** session/seq over the wire, forged-session rejection, sessions-flag independence, auth-cookie never read | done (15) |
|
||||
| `tests/web/test_api.py` | **v2:** miss text captured server-side only, hits never stored, never in the app access log | done (2) |
|
||||
|
||||
### Frontend (`frontend/`)
|
||||
|
||||
| File | Change | Status |
|
||||
|---|---|---|
|
||||
| `static/consent.js` | **v2, new** — opt-in banner, equal-weight buttons, localStorage consent record with re-ask intervals, GPC/DNT stand-down, footer withdrawal control, `onConsentChange` | done |
|
||||
| `static/track.js` | flag check, batching, `currentView()`, `[data-event]` delegation, `trackLegacy()`; **v2:** consent gate, session minting/rotation/caps, `seq`, immediate drop on withdrawal | done |
|
||||
| `static/style.css` | **v2:** `.consent-bar` / `.consent-control`, tokens only, 44px targets, mobile stacking | done |
|
||||
| `static/digest.js` | beacon moved out; re-exports for importers | done |
|
||||
| `static/app.js`, `mappicker.js`, `units.js` | `view.open`, `place.pick`, `deadend`, `share`, `view.control` call sites | done |
|
||||
| `static/*.html` (6 shells) | load `consent.js` then `track.js` | done |
|
||||
| `templates/base.html.j2` | flag stamp on `<html>`; **v2:** consent script, footer withdrawal control, "No third-party trackers" | done |
|
||||
| `templates/home.html.j2` | **v2:** hero trust line no longer claims "No tracking" | done |
|
||||
| `templates/privacy.html.j2` | **v2:** full rewrite (§7) | done |
|
||||
| `content.py`, `app.py` | `EVENTS_ENABLED` on both HTML paths | done |
|
||||
| `static/calendar.js`, `compare.js`, `day.js`, `score.js`, `subscriptions.js` | their own call sites | **todo** |
|
||||
| `tests/unit/test_pages.py` | flag off by default / stamped when on; **v2:** consent control present everywhere, script order, privacy-copy promises, footer claim | done (6) |
|
||||
|
||||
### Rollout
|
||||
|
||||
1. Land with **all three flags unset everywhere**. Inert: the beacon answers 204,
|
||||
no sink is written, no banner renders, no id is minted, and the flag stamp is
|
||||
absent from every page (asserted by test).
|
||||
2. **Ship the privacy-page copy and the IP truncation first** — including the
|
||||
Caddy log-format fix (§5.4). The page must not claim addresses are shortened
|
||||
before they are.
|
||||
3. Migrate + enable **Tier A only** on dev, then beta, then prod. Verify in
|
||||
Grafana (`tag="ui.event"`) that no free text or coordinate ever appears.
|
||||
4. Run Tier A for a month. Write down which questions it failed to answer (§6b).
|
||||
5. Only then, and only if step 4 produced surviving questions: enable
|
||||
`THERMOGRAPH_SEARCH_MISS` and `THERMOGRAPH_EVENT_SESSIONS` on **prod only**,
|
||||
as a time-boxed study. Watch the consent accept rate and compare Tier B's
|
||||
rates against Tier A's to quantify the bias before trusting any funnel.
|
||||
6. Roll up and turn Tier B off at the end of the window; the banner disappears
|
||||
with it.
|
||||
7. Kill switch: unset a flag. No deploy, no migration, no data loss — and
|
||||
unsetting `THERMOGRAPH_EVENT_SESSIONS` alone removes the banner and the
|
||||
identifier while leaving the anonymous history intact.
|
||||
|
||||
### Cost check
|
||||
|
||||
~3,000 human req/day → ~1–3k events/day, batched ≈ a few hundred extra
|
||||
requests/day (~0.005 req/s). Tier A grows a few hundred rows/day. Tier B, at a
|
||||
35% consent rate and ~10 events/session, is on the order of 2–3k rows/day held
|
||||
for 30 days — under a million rows at steady state, trivially small. `ui_search_miss`
|
||||
is bounded by distinct failed queries per day, realistically tens.
|
||||
69
backend/alembic/versions/0003_ui_events.py
Normal file
69
backend/alembic/versions/0003_ui_events.py
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
"""UI product events: hourly aggregate hypertable
|
||||
|
||||
One row per (hour, event, view, prop, value, referrer) — never one row per
|
||||
interaction. See ``backend/core/events.py`` for the schema and the reasoning:
|
||||
the table has no column an identifier could be written into, so it cannot hold
|
||||
a sequence of one person's actions even in principle.
|
||||
|
||||
TimescaleDB hypertable on ``bucket`` with a 30-day chunk interval (the read
|
||||
pattern is "last N weeks, grouped by event"), plus a retention policy so old
|
||||
buckets drop themselves. ``RETENTION_DAYS`` is the operator-set retention limit
|
||||
— change it here, not by hand on the box, so the deployed value is reviewable.
|
||||
|
||||
Postgres-only: guarded to no-op on the SQLite bind (tests / CI / offline
|
||||
Alembic), where events.py's aggregate sink is inert and only the JSONL activity
|
||||
line is written.
|
||||
|
||||
Revision ID: 0003_ui_events
|
||||
Revises: 0002_climate_hypertables
|
||||
Create Date: 2026-07-23
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
revision = "0003_ui_events"
|
||||
down_revision = "0002_climate_hypertables"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
# Aggregate rows carry no personal data, so this is a storage/usefulness choice
|
||||
# rather than a GDPR one. 400 days keeps a full year-over-year comparison.
|
||||
RETENTION_DAYS = 400
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
if op.get_bind().dialect.name != "postgresql":
|
||||
return # SQLite fallback (tests/CI): the aggregate sink is a no-op there.
|
||||
|
||||
op.execute("CREATE EXTENSION IF NOT EXISTS timescaledb")
|
||||
|
||||
# Every dimension is NOT NULL DEFAULT '' rather than nullable: '' is a real
|
||||
# value ("this event does not use this slot"), and it keeps the primary key
|
||||
# usable (NULLs would defeat the ON CONFLICT upsert).
|
||||
op.execute("""
|
||||
CREATE TABLE ui_event_hourly (
|
||||
bucket TIMESTAMPTZ NOT NULL,
|
||||
event TEXT NOT NULL,
|
||||
view TEXT NOT NULL DEFAULT '',
|
||||
prop TEXT NOT NULL DEFAULT '',
|
||||
value TEXT NOT NULL DEFAULT '',
|
||||
referrer TEXT NOT NULL DEFAULT '',
|
||||
count BIGINT NOT NULL DEFAULT 0,
|
||||
PRIMARY KEY (bucket, event, view, prop, value, referrer)
|
||||
)
|
||||
""")
|
||||
# The PK includes the range-partition column (bucket), which TimescaleDB
|
||||
# requires of any unique index on a hypertable.
|
||||
op.execute(
|
||||
"SELECT create_hypertable('ui_event_hourly', "
|
||||
"by_range('bucket', INTERVAL '30 days'))")
|
||||
op.execute(
|
||||
f"SELECT add_retention_policy('ui_event_hourly', INTERVAL '{RETENTION_DAYS} days')")
|
||||
|
||||
# The dashboard query is "one event over time", so index it that way.
|
||||
op.execute("CREATE INDEX ui_event_hourly_event_idx ON ui_event_hourly (event, bucket DESC)")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
if op.get_bind().dialect.name != "postgresql":
|
||||
return
|
||||
op.execute("DROP TABLE IF EXISTS ui_event_hourly")
|
||||
98
backend/alembic/versions/0004_ui_event_sessions.py
Normal file
98
backend/alembic/versions/0004_ui_event_sessions.py
Normal file
|
|
@ -0,0 +1,98 @@
|
|||
"""UI events tier B: consented session rows + failed-search text
|
||||
|
||||
Two tables, both created together because they arrived from the same pair of
|
||||
operator decisions (see UI-EVENTS.md §6):
|
||||
|
||||
* ``ui_event_session`` — one row per event, carrying an ephemeral per-tab session
|
||||
id, so a funnel can be computed. This is a **behavioural sequence dataset**:
|
||||
the thing ``ui_event_hourly`` was deliberately shaped to avoid. It therefore
|
||||
gets the shortest retention in the system (30 days), minute-granularity
|
||||
timestamps rather than precise ones, and an in-session ``seq`` to carry order.
|
||||
Only ever written for a visitor who gave opt-in consent.
|
||||
* ``ui_search_miss`` — normalised zero-result search text, the only free-text
|
||||
field anywhere in the instrumentation, captured server-side only. Stored as a
|
||||
per-day count rather than a row per search, and pruned below a k-anonymity
|
||||
floor (see ``core/events.py`` SEARCH_MISS_MIN_COUNT) so a string one person
|
||||
typed once does not persist.
|
||||
|
||||
Retention numbers live here, not in a hand-typed policy on the box, so they are
|
||||
reviewable in the repo and move with the code.
|
||||
|
||||
Postgres-only: guarded to no-op on the SQLite bind (tests / CI / offline
|
||||
Alembic), where both sinks are inert.
|
||||
|
||||
Revision ID: 0004_ui_event_sessions
|
||||
Revises: 0003_ui_events
|
||||
Create Date: 2026-07-23
|
||||
"""
|
||||
from alembic import op
|
||||
|
||||
revision = "0004_ui_event_sessions"
|
||||
down_revision = "0003_ui_events"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
# The behavioural-sequence table. Short on purpose: a funnel question is answered
|
||||
# in days-to-weeks, and the derived answer should be rolled up into the anonymous
|
||||
# aggregate rather than kept as raw sequences.
|
||||
SESSION_RETENTION_DAYS = 30
|
||||
|
||||
# Free-text search misses. Long enough to spot a seasonal gap in the place index,
|
||||
# short enough to be defensible for a field a person can type anything into.
|
||||
SEARCH_MISS_RETENTION_DAYS = 90
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
if op.get_bind().dialect.name != "postgresql":
|
||||
return # SQLite fallback (tests/CI): both sinks are no-ops there.
|
||||
|
||||
op.execute("CREATE EXTENSION IF NOT EXISTS timescaledb")
|
||||
|
||||
# No IP column, no user-agent column, no account-id column, and no unique
|
||||
# constraint that would require one: there is deliberately nothing here that
|
||||
# could re-identify a session, and nothing to join it to.
|
||||
op.execute("""
|
||||
CREATE TABLE ui_event_session (
|
||||
ts TIMESTAMPTZ NOT NULL,
|
||||
session TEXT NOT NULL,
|
||||
seq INTEGER NOT NULL,
|
||||
event TEXT NOT NULL,
|
||||
view TEXT NOT NULL DEFAULT '',
|
||||
prop TEXT NOT NULL DEFAULT '',
|
||||
value TEXT NOT NULL DEFAULT '',
|
||||
referrer TEXT NOT NULL DEFAULT ''
|
||||
)
|
||||
""")
|
||||
# 1-day chunks: the table is dropped a chunk at a time by the retention
|
||||
# policy, and every query against it is "the last N days".
|
||||
op.execute(
|
||||
"SELECT create_hypertable('ui_event_session', by_range('ts', INTERVAL '1 day'))")
|
||||
op.execute(
|
||||
f"SELECT add_retention_policy('ui_event_session', "
|
||||
f"INTERVAL '{SESSION_RETENTION_DAYS} days')")
|
||||
# The only access pattern is "walk one session in order" (funnels) and
|
||||
# "everything in a window" (the rollup).
|
||||
op.execute("CREATE INDEX ui_event_session_sid_idx ON ui_event_session (session, seq)")
|
||||
|
||||
# Failed searches: a per-day count, never a row per search.
|
||||
op.execute("""
|
||||
CREATE TABLE ui_search_miss (
|
||||
day DATE NOT NULL,
|
||||
q TEXT NOT NULL,
|
||||
count BIGINT NOT NULL DEFAULT 0,
|
||||
PRIMARY KEY (day, q)
|
||||
)
|
||||
""")
|
||||
# Plain table, not a hypertable: it is tiny (bounded by distinct failed
|
||||
# queries per day) and the prune is a predicate on `count`, not a chunk drop,
|
||||
# so time-partitioning buys nothing. Retention is enforced by the same daily
|
||||
# job that applies the k-anonymity prune -- see core/events.py
|
||||
# PRUNE_SEARCH_MISS_SQL and UI-EVENTS.md.
|
||||
op.execute("CREATE INDEX ui_search_miss_day_idx ON ui_search_miss (day)")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
if op.get_bind().dialect.name != "postgresql":
|
||||
return
|
||||
op.execute("DROP TABLE IF EXISTS ui_search_miss")
|
||||
op.execute("DROP TABLE IF EXISTS ui_event_session")
|
||||
526
backend/core/events.py
Normal file
526
backend/core/events.py
Normal file
|
|
@ -0,0 +1,526 @@
|
|||
"""Product-event instrumentation: what people actually *do* in the UI.
|
||||
|
||||
This is the durable, dimensioned tier that sits above ``core/metrics.py``'s
|
||||
since-start counters. metrics.py answers "is traffic flowing right now"; this
|
||||
module answers "did anyone ever get a location, and did search work" — over
|
||||
weeks, across deploys.
|
||||
|
||||
**Two tiers, two flags, both OFF by default.** Everything here is inert unless
|
||||
the matching flag is set, so shipping it is a no-op.
|
||||
|
||||
* **Tier A — anonymous aggregates (``THERMOGRAPH_EVENTS``).** Hourly counts keyed
|
||||
by allowlisted enums. No identifier of any kind, nothing written to the
|
||||
visitor's device, so it needs no consent. This runs for *every* visitor and is
|
||||
the unbiased signal.
|
||||
* **Tier B — consented session rows (``THERMOGRAPH_EVENT_SESSIONS``).** The same
|
||||
events, additionally carrying an ephemeral per-tab session id, written one row
|
||||
per event so a funnel ("of the sessions that opened the map, how many reached a
|
||||
grade?") can actually be computed. **Only ever sent by a browser whose visitor
|
||||
gave opt-in consent** — see frontend/static/consent.js. This is a behavioural
|
||||
sequence dataset, which is exactly what Tier A was designed to avoid, so it
|
||||
carries a much shorter retention and extra minimisation.
|
||||
|
||||
Tier B depends on Tier A: with ``THERMOGRAPH_EVENTS`` off, nothing records at all.
|
||||
|
||||
Design constraints, in priority order:
|
||||
|
||||
1. **No stable identifier.** No cookie, no cross-session id, no fingerprint, and
|
||||
never the account id — see ``validate_session``. The Tier B id is minted in
|
||||
the browser, lives in ``sessionStorage`` (per tab), rotates on a 30-minute idle
|
||||
/ 2-hour absolute cap, and is unlinkable to any other session, device, or
|
||||
account. Tier A has no id column at all.
|
||||
2. **No free text, no coordinates.** Every event field is drawn from a fixed
|
||||
allowlist (see ``SCHEMA``); an unrecognised name or value collapses to a
|
||||
bounded "other" bucket. The one deliberate exception is the failed-search text
|
||||
captured server-side by ``record_search_miss`` — heavily normalised, rejected
|
||||
on any personal-data smell, and pruned unless several people typed the same
|
||||
thing. It is never accepted from the browser.
|
||||
3. **No IP.** The caller's IP is used only as an in-memory rate-limit key for the
|
||||
current minute (see ``metrics.rate_ok``) and is never passed in here, never
|
||||
stored, and never joined to a session id.
|
||||
4. **Never breaks a request.** Every entry point swallows its own errors.
|
||||
|
||||
Storage (Postgres; see ``alembic/versions/0003_ui_events.py`` and ``0004``):
|
||||
|
||||
* ``ui_event_hourly`` — Tier A hourly aggregate, 400-day retention.
|
||||
* ``ui_event_session`` — Tier B per-event rows, **30-day** retention, minute-
|
||||
granularity timestamps and an in-session sequence number instead of precise
|
||||
clock times.
|
||||
* ``ui_search_miss`` — normalised zero-result queries, 90-day retention, with
|
||||
singletons pruned at 7 days (see ``SEARCH_MISS_MIN_COUNT``).
|
||||
|
||||
Plus one JSONL line per event into the activity stream so Loki/Grafana can show
|
||||
the live picture. Off Postgres (dev, tests, offline tooling) the SQL sinks
|
||||
degrade to no-ops and only the JSONL line is written — the same dialect split
|
||||
``data/store.py`` and ``core/metrics.py`` already use.
|
||||
"""
|
||||
import datetime
|
||||
import os
|
||||
import re
|
||||
import threading
|
||||
import unicodedata
|
||||
|
||||
from core import audit
|
||||
|
||||
# --- feature flags -------------------------------------------------------------
|
||||
# Default OFF everywhere. Turn on per-environment (dev first, then beta, then
|
||||
# prod) once the privacy copy in frontend/templates/privacy.html.j2 matches what
|
||||
# is actually recorded.
|
||||
|
||||
|
||||
def _flag(name: str) -> bool:
|
||||
return os.environ.get(name, "").strip().lower() in ("1", "true", "yes", "on")
|
||||
|
||||
|
||||
#: Tier A. Nothing at all records without this.
|
||||
ENABLED = _flag("THERMOGRAPH_EVENTS")
|
||||
|
||||
#: Tier B. Consented, session-scoped rows. Separate flag on purpose: Tier A is
|
||||
#: the thing worth running permanently, and Tier B should be switchable off
|
||||
#: (e.g. at the end of a time-boxed study) without losing the aggregate history.
|
||||
SESSIONS_ENABLED = _flag("THERMOGRAPH_EVENT_SESSIONS")
|
||||
|
||||
#: Zero-result search text. Separate flag again — it is the only free-text field
|
||||
#: in the system and deserves its own on/off switch and its own decision.
|
||||
SEARCH_MISS_ENABLED = _flag("THERMOGRAPH_SEARCH_MISS")
|
||||
|
||||
DATABASE_URL = os.environ.get("THERMOGRAPH_DATABASE_URL", "").strip()
|
||||
IS_POSTGRES = DATABASE_URL.startswith("postgresql")
|
||||
|
||||
|
||||
def _libpq_dsn(url: str) -> str:
|
||||
"""A plain libpq URL psycopg accepts (mirrors climate_store/metrics)."""
|
||||
return url.replace("+asyncpg", "").replace("+psycopg", "")
|
||||
|
||||
|
||||
_PG_DSN = _libpq_dsn(DATABASE_URL) if IS_POSTGRES else ""
|
||||
|
||||
# --- schema -------------------------------------------------------------------
|
||||
# Seven events. Every one answers a product question that cannot be answered from
|
||||
# the request log alone; anything the request log *can* answer (page views, API
|
||||
# usage, error rates) is deliberately NOT duplicated here.
|
||||
#
|
||||
# Three dimension slots are stored — ``view``, ``prop``, ``value`` — plus a
|
||||
# referrer domain on ``view.open`` only (acquisition), so the key space stays
|
||||
# small enough to eyeball. Each event declares which slots it uses and the exact
|
||||
# allowlist for each.
|
||||
|
||||
VIEWS = frozenset({
|
||||
"home", "calendar", "day", "compare", "score", "alerts", "legend",
|
||||
"city", "cities", "records", "month", "glossary", "about", "privacy",
|
||||
})
|
||||
|
||||
#: Where a location came from. The activation question: do visitors ever get
|
||||
#: past "no place picked", and by which route?
|
||||
PICK_METHODS = frozenset({"geolocate", "map", "search", "restore", "link"})
|
||||
|
||||
#: Did the location search actually work? ``hit`` = results shown, ``typo`` =
|
||||
#: results only after respelling (places.suggest's ``corrected``), ``miss`` =
|
||||
#: nothing found. The query string itself is never sent or stored.
|
||||
SEARCH_OUTCOMES = frozenset({"hit", "typo", "miss"})
|
||||
|
||||
#: In-view controls worth knowing about. Anything not listed is dropped, so a
|
||||
#: new control has to be added here deliberately (and reviewed for privacy).
|
||||
CONTROLS = frozenset({
|
||||
"chart_metric", # value: the metric key
|
||||
"unit", # value: F | C
|
||||
"date", # value: today | past
|
||||
"range", # value: the span bucket
|
||||
"season", # value: season key or "custom"
|
||||
"totals", # calendar totals mode
|
||||
"dist_metric", # compare distribution metric
|
||||
"comfort", # compare comfort slicer moved
|
||||
"compare_add", # value: the location-count bucket after the add
|
||||
"compare_remove",
|
||||
"filter_sheet", # mobile filter sheet opened
|
||||
"day_step", # day view prev/next
|
||||
})
|
||||
|
||||
#: Bounded value vocabulary, shared across controls. A value outside this set is
|
||||
#: stored as "other" — never verbatim.
|
||||
VALUES = frozenset({
|
||||
# chart / distribution metrics
|
||||
"tmax", "tmin", "feels", "humid", "wind", "gust", "precip", "dry",
|
||||
# units
|
||||
"F", "C",
|
||||
# dates
|
||||
"today", "past", "future",
|
||||
# spans / counts (bucketed client-side, never a raw number)
|
||||
"1", "2", "3", "4", "5plus",
|
||||
"1y", "2y", "5y", "10y", "max",
|
||||
# seasons
|
||||
"winter", "spring", "summer", "fall", "custom", "all",
|
||||
# generic
|
||||
"on", "off", "other",
|
||||
})
|
||||
|
||||
SHARE_KINDS = frozenset({"link", "png", "permalink"})
|
||||
|
||||
#: The alerts funnel. ``ok`` records whether the step succeeded, so a high
|
||||
#: start/low create ratio is visible without any per-person tracking.
|
||||
ALERT_STEPS = frozenset({"open", "start", "create", "push_on", "push_off", "test"})
|
||||
|
||||
#: Dead ends — the highest-signal events here. A spike in any of these is a bug
|
||||
#: report nobody filed.
|
||||
DEADENDS = frozenset({
|
||||
"grade_error", # /api/grade returned an error the user saw
|
||||
"no_data", # a cell with no usable history
|
||||
"geo_denied", # browser geolocation refused / timed out
|
||||
"search_miss", # search returned nothing (paired with place.search=miss)
|
||||
"not_found", # a 404 page was rendered
|
||||
"offline", # the fetch failed with no network
|
||||
})
|
||||
|
||||
OK = frozenset({"y", "n"})
|
||||
|
||||
#: name -> {slot: allowlist}. A slot absent from an event is always stored "".
|
||||
SCHEMA: "dict[str, dict[str, frozenset]]" = {
|
||||
# A view was opened (page load or SPA view). ``prop`` is how it was reached.
|
||||
"view.open": {"view": VIEWS, "prop": frozenset({"direct", "nav", "link", "restore"})},
|
||||
# A location was chosen. NEVER carries coordinates — only the route taken.
|
||||
"place.pick": {"view": VIEWS, "prop": PICK_METHODS},
|
||||
# A location search completed. Recorded server-side from api_suggest as well
|
||||
# as client-side, so searches from every surface are covered.
|
||||
"place.search": {"prop": SEARCH_OUTCOMES},
|
||||
# An in-view control was used.
|
||||
"view.control": {"view": VIEWS, "prop": CONTROLS, "value": VALUES},
|
||||
# Share / export.
|
||||
"share": {"view": VIEWS, "prop": SHARE_KINDS},
|
||||
# The alerts funnel.
|
||||
"alert": {"prop": ALERT_STEPS, "value": OK},
|
||||
# A dead end the visitor actually hit.
|
||||
"deadend": {"view": VIEWS, "prop": DEADENDS},
|
||||
}
|
||||
|
||||
#: Anything unrecognised collapses here with every dimension blank — visible
|
||||
#: enough to notice abuse, bounded enough to be harmless.
|
||||
EVENT_OTHER = "other"
|
||||
VALUE_OTHER = "other"
|
||||
|
||||
SLOTS = ("view", "prop", "value")
|
||||
|
||||
#: Max events per batched beacon payload. The client batches bursts (chart-metric
|
||||
#: taps, slider drags) rather than firing a request each.
|
||||
MAX_BATCH = 20
|
||||
|
||||
#: Max JSON body the beacon will read, in bytes. Must comfortably exceed
|
||||
#: MAX_BATCH full-sized events (~120 bytes each with a session id) or the byte
|
||||
#: cap, not the batch cap, becomes the binding limit and a legitimate full batch
|
||||
#: is silently dropped. Still far too small to be worth abusing as storage.
|
||||
MAX_BODY_BYTES = 4096
|
||||
|
||||
# --- Tier B: the ephemeral session id ------------------------------------------
|
||||
# The id is minted in the browser (16 random bytes, base64url, 22 chars) and is
|
||||
# the ONE field a client could otherwise use to smuggle arbitrary text into the
|
||||
# pipeline — a free-text column with a different name. So it is validated by
|
||||
# shape, not merely length-capped: anything that is not exactly 22 base64url
|
||||
# characters is discarded and the event still records to Tier A. There is
|
||||
# nothing to gain by sending a "clever" id, and nothing gets stored if you try.
|
||||
_SESSION_RE = re.compile(r"\A[A-Za-z0-9_-]{22}\Z")
|
||||
|
||||
#: Rows per session before Tier B stops recording (Tier A keeps counting). Bounds
|
||||
#: both storage and how long/detailed any one behavioural sequence can get — a
|
||||
#: 200-step sequence is already far more than any funnel question needs.
|
||||
MAX_SEQ = 200
|
||||
|
||||
|
||||
def validate_session(sid) -> str:
|
||||
"""A well-formed ephemeral session id, or "" — never anything else.
|
||||
|
||||
Deliberately NOT a sanitiser: an id that does not match the expected shape is
|
||||
dropped entirely rather than trimmed into shape, because the only thing a
|
||||
malformed id can be is either a bug or an attempt to use the field as a
|
||||
free-text channel, and both should lose their payload completely.
|
||||
|
||||
The account id is never accepted here and is never read on this route: see
|
||||
web/app.py's api_event, which does not touch the session cookie. Tying an
|
||||
anonymous behavioural sequence to an identified person would convert this
|
||||
whole dataset into personal data about a known individual, for essentially
|
||||
no analytical gain (the questions are about the product, not about who).
|
||||
"""
|
||||
if not isinstance(sid, str):
|
||||
return ""
|
||||
return sid if _SESSION_RE.match(sid) else ""
|
||||
|
||||
|
||||
def validate_seq(seq) -> int:
|
||||
"""An in-session sequence number in [0, MAX_SEQ], or -1 for "not in a
|
||||
session". Ordering is carried by this counter rather than by a precise
|
||||
timestamp, so the stored time can stay coarse (see ``_minute_bucket``)."""
|
||||
if isinstance(seq, bool) or not isinstance(seq, int):
|
||||
return -1
|
||||
return seq if 0 <= seq <= MAX_SEQ else -1
|
||||
|
||||
|
||||
# --- the one free-text field: zero-result search queries ------------------------
|
||||
# Captured server-side only (api_suggest), never accepted from a browser. What
|
||||
# people searched for and did NOT find is the single highest-value product signal
|
||||
# here — it names the gaps in the place index — but a search box is a field a
|
||||
# person can type anything into, so this is the only place where personal data
|
||||
# could arrive incidentally. Four layers of mitigation, in order of strength:
|
||||
#
|
||||
# 1. Only ZERO-RESULT queries are considered at all. A query that found a place
|
||||
# is already answered by the `hit` counter and is never stored as text.
|
||||
# 2. Reject (never sanitise) anything with a personal-data smell — an address,
|
||||
# an email, a URL, a long digit run, a sentence. Stripping bad characters
|
||||
# would keep the text; rejecting loses it, which is the point.
|
||||
# 3. Store as a COUNT per normalised string per day, not as a row per search.
|
||||
# 4. Prune anything only ever typed by a couple of people (see
|
||||
# SEARCH_MISS_MIN_COUNT). A string three different people typed is a product
|
||||
# signal; a string one person typed once is the risky case, and it is the
|
||||
# one that gets deleted.
|
||||
|
||||
#: Longer than this is a sentence or an address, not a place name.
|
||||
SEARCH_MISS_MAX_LEN = 40
|
||||
#: More words than this likewise.
|
||||
SEARCH_MISS_MAX_WORDS = 5
|
||||
#: A run of digits this long is a phone number / ID, not a postcode.
|
||||
SEARCH_MISS_MAX_DIGIT_RUN = 6
|
||||
#: Rows below this count are deleted once they are older than
|
||||
#: SEARCH_MISS_PRUNE_DAYS — a k-anonymity floor on the free-text field.
|
||||
SEARCH_MISS_MIN_COUNT = 3
|
||||
SEARCH_MISS_PRUNE_DAYS = 7
|
||||
|
||||
# Letters (any script), marks, digits, and the punctuation that genuinely occurs
|
||||
# in place names. Anything else is a smell, not a typo.
|
||||
_PLACE_OK = re.compile(r"\A[^\W_]+(?:[ \-'.,()·’][^\W_]+)*\Z", re.UNICODE)
|
||||
_DIGIT_RUN = re.compile(r"\d{%d,}" % (SEARCH_MISS_MAX_DIGIT_RUN + 1))
|
||||
_URLISH = re.compile(r"(://|www\.|\.[a-z]{2,}(/|\Z))", re.IGNORECASE)
|
||||
|
||||
|
||||
def normalize_search_miss(q) -> str:
|
||||
"""A failed query reduced to a storable form, or "" to store nothing.
|
||||
|
||||
NFKC-normalised, casefolded, whitespace-collapsed, length- and word-capped.
|
||||
Returns "" — meaning *drop it* — for anything that smells like personal data
|
||||
rather than a place name.
|
||||
"""
|
||||
if not isinstance(q, str):
|
||||
return ""
|
||||
q = unicodedata.normalize("NFKC", q).strip()
|
||||
q = " ".join(q.split()) # collapse all internal whitespace runs
|
||||
if not q:
|
||||
return ""
|
||||
# Casefold, not lower(): correct for ß/İ and friends, and it collapses the
|
||||
# trivial "same query, different capitalisation" variants into one row.
|
||||
q = q.casefold()
|
||||
if len(q) > SEARCH_MISS_MAX_LEN:
|
||||
return ""
|
||||
if len(q.split(" ")) > SEARCH_MISS_MAX_WORDS:
|
||||
return ""
|
||||
if "@" in q or _URLISH.search(q): # email / URL / hostname
|
||||
return ""
|
||||
if _DIGIT_RUN.search(q): # phone number, account id, coords
|
||||
return ""
|
||||
if not _PLACE_OK.match(q): # punctuation soup, not a place name
|
||||
return ""
|
||||
return q
|
||||
|
||||
|
||||
def normalize(name: str, props: "dict | None" = None) -> "tuple[str, dict[str, str]]":
|
||||
"""An allowlisted ``(event, {view, prop, value})`` pair.
|
||||
|
||||
The allowlist is the whole security model of an unauthenticated write
|
||||
endpoint: it is what stops the beacon from becoming unbounded storage, a
|
||||
spam sink, or an accidental PII channel. Unknown event -> ``EVENT_OTHER``
|
||||
with blank dimensions; unknown value in a known slot -> ``VALUE_OTHER``;
|
||||
a slot the event does not declare -> dropped entirely.
|
||||
"""
|
||||
name = (name or "").strip()
|
||||
spec = SCHEMA.get(name)
|
||||
if spec is None:
|
||||
return EVENT_OTHER, {s: "" for s in SLOTS}
|
||||
props = props if isinstance(props, dict) else {}
|
||||
out: "dict[str, str]" = {}
|
||||
for slot in SLOTS:
|
||||
allowed = spec.get(slot)
|
||||
if allowed is None:
|
||||
out[slot] = ""
|
||||
continue
|
||||
raw = props.get(slot)
|
||||
raw = raw.strip() if isinstance(raw, str) else ""
|
||||
out[slot] = raw if raw in allowed else (VALUE_OTHER if raw else "")
|
||||
return name, out
|
||||
|
||||
|
||||
# --- durable aggregate sink ---------------------------------------------------
|
||||
# One row per (hour, event, view, prop, value, referrer). No row is ever written
|
||||
# per interaction, so the table cannot hold a sequence of one person's actions
|
||||
# even in principle.
|
||||
|
||||
_local = threading.local()
|
||||
|
||||
|
||||
def _conn():
|
||||
"""Thread-local autocommit psycopg connection, or None when Postgres is off
|
||||
or unreachable. Mirrors data/climate_store.py."""
|
||||
if not IS_POSTGRES:
|
||||
return None
|
||||
conn = getattr(_local, "conn", None)
|
||||
if conn is not None and not conn.closed:
|
||||
return conn
|
||||
try:
|
||||
import psycopg # local import: only the Postgres path needs it
|
||||
_local.conn = psycopg.connect(_PG_DSN, autocommit=True)
|
||||
except Exception: # noqa: BLE001 - degrade to "no durable sink", never raise
|
||||
_local.conn = None
|
||||
return _local.conn
|
||||
|
||||
|
||||
def _hour_bucket() -> datetime.datetime:
|
||||
now = datetime.datetime.now(datetime.timezone.utc)
|
||||
return now.replace(minute=0, second=0, microsecond=0)
|
||||
|
||||
|
||||
def _minute_bucket() -> datetime.datetime:
|
||||
"""Tier B timestamps are truncated to the minute. Precise clock times are a
|
||||
fingerprint (inter-event millisecond gaps are individuating, and they let two
|
||||
rotated sessions be stitched back together by adjacency); the *order* of a
|
||||
session's events is what a funnel needs, and that is carried by ``seq``."""
|
||||
now = datetime.datetime.now(datetime.timezone.utc)
|
||||
return now.replace(second=0, microsecond=0)
|
||||
|
||||
|
||||
def _utc_day() -> datetime.date:
|
||||
return datetime.datetime.now(datetime.timezone.utc).date()
|
||||
|
||||
|
||||
_UPSERT = """
|
||||
INSERT INTO ui_event_hourly (bucket, event, view, prop, value, referrer, count)
|
||||
VALUES (%s, %s, %s, %s, %s, %s, 1)
|
||||
ON CONFLICT (bucket, event, view, prop, value, referrer)
|
||||
DO UPDATE SET count = ui_event_hourly.count + 1
|
||||
"""
|
||||
|
||||
|
||||
def _upsert(event: str, dims: "dict[str, str]", referrer: str) -> None:
|
||||
conn = _conn()
|
||||
if conn is None:
|
||||
return
|
||||
try:
|
||||
conn.execute(_UPSERT, (_hour_bucket(), event, dims["view"], dims["prop"],
|
||||
dims["value"], referrer))
|
||||
except Exception: # noqa: BLE001 - a broken connection retires itself below
|
||||
try:
|
||||
conn.close()
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
_local.conn = None
|
||||
|
||||
|
||||
_INSERT_SESSION = """
|
||||
INSERT INTO ui_event_session (ts, session, seq, event, view, prop, value, referrer)
|
||||
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
|
||||
"""
|
||||
|
||||
|
||||
def _insert_session(session: str, seq: int, event: str,
|
||||
dims: "dict[str, str]", referrer: str) -> None:
|
||||
conn = _conn()
|
||||
if conn is None:
|
||||
return
|
||||
try:
|
||||
conn.execute(_INSERT_SESSION, (_minute_bucket(), session, seq, event,
|
||||
dims["view"], dims["prop"], dims["value"], referrer))
|
||||
except Exception: # noqa: BLE001 - a broken connection retires itself
|
||||
try:
|
||||
conn.close()
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
_local.conn = None
|
||||
|
||||
|
||||
_UPSERT_SEARCH_MISS = """
|
||||
INSERT INTO ui_search_miss (day, q, count) VALUES (%s, %s, 1)
|
||||
ON CONFLICT (day, q) DO UPDATE SET count = ui_search_miss.count + 1
|
||||
"""
|
||||
|
||||
|
||||
def _upsert_search_miss(q: str) -> None:
|
||||
conn = _conn()
|
||||
if conn is None:
|
||||
return
|
||||
try:
|
||||
conn.execute(_UPSERT_SEARCH_MISS, (_utc_day(), q))
|
||||
except Exception: # noqa: BLE001
|
||||
try:
|
||||
conn.close()
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
_local.conn = None
|
||||
|
||||
|
||||
#: Deletes the free-text rows that only one or two people ever produced, once
|
||||
#: they are old enough that the count has stopped growing. Run daily (see the
|
||||
#: rollout section of UI-EVENTS.md); parameterised here rather than typed by hand
|
||||
#: on the box so the retention rule is reviewable in the repo.
|
||||
PRUNE_SEARCH_MISS_SQL = (
|
||||
"DELETE FROM ui_search_miss "
|
||||
"WHERE day < CURRENT_DATE - INTERVAL '%d days' AND count < %d"
|
||||
% (SEARCH_MISS_PRUNE_DAYS, SEARCH_MISS_MIN_COUNT)
|
||||
)
|
||||
|
||||
|
||||
# --- public entry points -------------------------------------------------------
|
||||
|
||||
def record(name: str, props: "dict | None" = None,
|
||||
referer: "str | None" = None, host: "str | None" = None,
|
||||
session: "str | None" = None, seq=None) -> None:
|
||||
"""Record one product event. Best-effort, silent, and a no-op unless the
|
||||
feature flag is on.
|
||||
|
||||
``referer``/``host`` come from the request's own headers (a client-supplied
|
||||
referrer would be forgeable and buys nothing) and are reduced to a bare
|
||||
domain by ``metrics.normalize_referrer`` — never a full URL, which can carry
|
||||
a search query or a private path. The referrer dimension is kept only for
|
||||
``view.open``; every other event stores "" there, which is what keeps the
|
||||
key space small.
|
||||
|
||||
``session``/``seq`` are the Tier B additions. They are only ever populated by
|
||||
a browser whose visitor opted in, they are validated by shape (see
|
||||
``validate_session``), and Tier A records identically whether they are
|
||||
present or not — so a dropped or refused session id costs the aggregate
|
||||
nothing.
|
||||
"""
|
||||
if not ENABLED:
|
||||
return
|
||||
try:
|
||||
# Import here, not at module scope: metrics imports cleanly on its own and
|
||||
# this keeps the dependency one-directional at import time.
|
||||
from core import metrics
|
||||
event, dims = normalize(name, props)
|
||||
referrer = metrics.normalize_referrer(referer, host) if event == "view.open" else ""
|
||||
# Tier A first and unconditionally: the anonymous aggregate is the signal
|
||||
# that must never depend on consent rates, client behaviour, or Tier B.
|
||||
_upsert(event, dims, referrer)
|
||||
sid = validate_session(session) if SESSIONS_ENABLED else ""
|
||||
n = validate_seq(seq) if sid else -1
|
||||
if sid and n >= 0:
|
||||
_insert_session(sid, n, event, dims, referrer)
|
||||
# Second sink: one structured line per event into the activity stream, so
|
||||
# the Loki/Grafana stack can show the live picture without waiting on an
|
||||
# hourly rollup. Carries the same allowlisted enums and nothing else —
|
||||
# NOT the session id, which would put a behavioural sequence into a log
|
||||
# stream with its own retention and its own access rules.
|
||||
audit.log_activity("ui.event", {"event": event, **dims, "referrer": referrer})
|
||||
except Exception: # noqa: BLE001 - instrumentation must never break a request
|
||||
pass
|
||||
|
||||
|
||||
def record_search_miss(q) -> None:
|
||||
"""Record one zero-result search query, server-side only.
|
||||
|
||||
Never call this with a query that returned results, and never accept the
|
||||
string from a browser: the browser sending its own search text would put it
|
||||
in a request body, in the client, and in anything that logs bodies — all of
|
||||
which this deliberately avoids. Silently stores nothing when the query does
|
||||
not survive ``normalize_search_miss``.
|
||||
"""
|
||||
if not (ENABLED and SEARCH_MISS_ENABLED):
|
||||
return
|
||||
try:
|
||||
norm = normalize_search_miss(q)
|
||||
if norm:
|
||||
_upsert_search_miss(norm)
|
||||
except Exception: # noqa: BLE001 - instrumentation must never break a request
|
||||
pass
|
||||
|
|
@ -705,12 +705,38 @@ def record_outbound(phase: str, outcome: str) -> None:
|
|||
|
||||
# Per-IP token bucket for the unauthenticated event beacon. Bounded: the map is
|
||||
# cleared whenever the minute rolls, so it holds at most one minute of clients.
|
||||
_EVENT_RATE_PER_MIN = 30
|
||||
#
|
||||
# 30/min was sized for the four coarse home.* counters (at most a handful per
|
||||
# visit). Interaction-level events (chart-metric taps, a comfort slider drag on
|
||||
# the compare page) are burstier by an order of magnitude, and a batched beacon
|
||||
# spends one token per event in the batch — at 30 the limiter silently truncates
|
||||
# ordinary use, which is worse than useless because the loss is invisible and
|
||||
# biased toward the most engaged visitors. 120/min is still a hard ceiling on
|
||||
# what one IP can write per minute. Env-tunable so it can be tightened on the
|
||||
# box without a deploy if the endpoint is ever abused.
|
||||
_EVENT_RATE_PER_MIN = int(os.environ.get("THERMOGRAPH_EVENT_RATE_PER_MIN", "120") or 120)
|
||||
_rate_lock = threading.Lock()
|
||||
_rate_minute = -1
|
||||
_rate_counts: "dict[str, int]" = {}
|
||||
|
||||
|
||||
def reset_rate_limiter() -> None:
|
||||
"""Drop every token bucket. For tests — one test's beacon traffic must not
|
||||
rate-limit the next one's, since TestClient always presents the same IP."""
|
||||
global _rate_minute
|
||||
with _rate_lock:
|
||||
_rate_minute = -1
|
||||
_rate_counts.clear()
|
||||
|
||||
|
||||
def rate_ok(ip: "str | None") -> bool:
|
||||
"""Public form of the beacon's per-IP token bucket, so a caller that fans one
|
||||
event out to several sinks (see web/app.py's ``_record_event``) spends exactly
|
||||
one token instead of one per sink. The IP is only ever a key in a map that is
|
||||
cleared every minute — it is never persisted anywhere."""
|
||||
return _rate_ok(ip)
|
||||
|
||||
|
||||
def _rate_ok(ip: "str | None") -> bool:
|
||||
global _rate_minute
|
||||
key = ip or "?"
|
||||
|
|
@ -725,13 +751,17 @@ def _rate_ok(ip: "str | None") -> bool:
|
|||
|
||||
|
||||
def record_event(name: str, referer: "str | None" = None,
|
||||
host: "str | None" = None, ip: "str | None" = None) -> None:
|
||||
host: "str | None" = None, ip: "str | None" = None,
|
||||
rate_limit: bool = True) -> None:
|
||||
"""Count one product event. The name is allowlisted and the referrer is taken
|
||||
from the request's own Referer header (never client-supplied, which would be
|
||||
trivially forgeable). Best-effort and silent: the caller always answers 204,
|
||||
so a rejected or rate-limited event tells a prober nothing."""
|
||||
so a rejected or rate-limited event tells a prober nothing.
|
||||
|
||||
``rate_limit=False`` skips the per-IP bucket for a caller that already spent
|
||||
the token through ``rate_ok`` — see web/app.py's ``_record_event``."""
|
||||
try:
|
||||
if not _rate_ok(ip):
|
||||
if rate_limit and not _rate_ok(ip):
|
||||
return
|
||||
event = normalize_event(name)
|
||||
# An unrecognized event carries no referrer dimension — one bucket total.
|
||||
|
|
|
|||
249
backend/tests/core/test_events.py
Normal file
249
backend/tests/core/test_events.py
Normal file
|
|
@ -0,0 +1,249 @@
|
|||
"""Unit tests for the UI product-event schema (core/events.py).
|
||||
|
||||
The allowlist in ``events.SCHEMA`` is the entire security and privacy model of an
|
||||
unauthenticated write endpoint, so most of what is worth asserting here is what
|
||||
does NOT get through: free text, coordinates, unknown names, unknown values.
|
||||
"""
|
||||
import importlib
|
||||
|
||||
from core import events
|
||||
|
||||
|
||||
def _fresh(monkeypatch, enabled=True):
|
||||
m = importlib.reload(events)
|
||||
monkeypatch.setattr(m, "ENABLED", enabled)
|
||||
return m
|
||||
|
||||
|
||||
def test_known_event_keeps_only_declared_slots(monkeypatch):
|
||||
m = _fresh(monkeypatch)
|
||||
name, dims = m.normalize("view.open", {"view": "calendar", "prop": "nav", "value": "tmax"})
|
||||
assert name == "view.open"
|
||||
# view.open declares view + prop but not value, so value is dropped entirely
|
||||
# rather than stored — an undeclared slot can never carry anything.
|
||||
assert dims == {"view": "calendar", "prop": "nav", "value": ""}
|
||||
|
||||
|
||||
def test_unknown_event_collapses_with_blank_dimensions(monkeypatch):
|
||||
m = _fresh(monkeypatch)
|
||||
name, dims = m.normalize("evil.event", {"view": "calendar", "prop": "nav"})
|
||||
assert name == m.EVENT_OTHER
|
||||
assert dims == {"view": "", "prop": "", "value": ""}
|
||||
|
||||
|
||||
def test_unknown_value_in_a_known_slot_is_bucketed_never_stored(monkeypatch):
|
||||
"""The property that makes the endpoint safe: nothing a client sends is ever
|
||||
persisted verbatim."""
|
||||
m = _fresh(monkeypatch)
|
||||
_, dims = m.normalize("place.pick", {"view": "home", "prop": "47.6,-122.3"})
|
||||
assert dims["prop"] == m.VALUE_OTHER
|
||||
_, dims = m.normalize("view.control", {"view": "home", "prop": "chart_metric",
|
||||
"value": "seattle wa"})
|
||||
assert dims["value"] == m.VALUE_OTHER
|
||||
|
||||
|
||||
def test_non_string_props_do_not_crash_or_leak(monkeypatch):
|
||||
m = _fresh(monkeypatch)
|
||||
for props in (None, [], "x", {"view": 47.6}, {"view": {"lat": 1}}):
|
||||
name, dims = m.normalize("place.pick", props)
|
||||
assert name == "place.pick"
|
||||
assert set(dims) == {"view", "prop", "value"}
|
||||
assert all(isinstance(v, str) for v in dims.values())
|
||||
|
||||
|
||||
def test_schema_slots_are_all_recognised(monkeypatch):
|
||||
"""Guard against a typo'd slot name silently making an event dimensionless."""
|
||||
m = _fresh(monkeypatch)
|
||||
for name, spec in m.SCHEMA.items():
|
||||
assert spec, name
|
||||
assert set(spec) <= set(m.SLOTS), name
|
||||
for slot, allowed in spec.items():
|
||||
assert isinstance(allowed, frozenset) and allowed, (name, slot)
|
||||
|
||||
|
||||
def test_no_slot_can_hold_free_text_by_construction(monkeypatch):
|
||||
"""Every allowlisted value is a short, boring token. If this ever fails,
|
||||
something person-identifying has been added to the vocabulary."""
|
||||
m = _fresh(monkeypatch)
|
||||
for spec in m.SCHEMA.values():
|
||||
for allowed in spec.values():
|
||||
for v in allowed:
|
||||
assert v.replace("_", "").replace("-", "").isalnum(), v
|
||||
assert len(v) <= 20, v
|
||||
|
||||
|
||||
def test_record_is_inert_when_the_flag_is_off(monkeypatch):
|
||||
m = _fresh(monkeypatch, enabled=False)
|
||||
calls = []
|
||||
monkeypatch.setattr(m, "_upsert", lambda *a: calls.append(a))
|
||||
monkeypatch.setattr(m.audit, "log_activity", lambda *a, **k: calls.append(a))
|
||||
m.record("view.open", {"view": "home"})
|
||||
assert calls == []
|
||||
|
||||
|
||||
def test_record_writes_one_activity_line_with_only_allowlisted_fields(monkeypatch):
|
||||
m = _fresh(monkeypatch)
|
||||
lines = []
|
||||
monkeypatch.setattr(m, "_upsert", lambda *a: None)
|
||||
monkeypatch.setattr(m.audit, "log_activity", lambda tag, rec: lines.append((tag, rec)))
|
||||
m.record("view.open", {"view": "city", "prop": "nav"},
|
||||
referer="https://news.ycombinator.com/item?id=1", host="thermograph.org")
|
||||
(tag, rec), = lines
|
||||
assert tag == "ui.event"
|
||||
# The referrer is reduced to a bare domain — never the URL, which can carry a
|
||||
# search query or a private path.
|
||||
assert rec == {"event": "view.open", "view": "city", "prop": "nav",
|
||||
"value": "", "referrer": "news.ycombinator.com"}
|
||||
|
||||
|
||||
def test_referrer_is_kept_only_for_view_open(monkeypatch):
|
||||
"""Acquisition needs it; nothing else does, and dropping it everywhere else is
|
||||
what keeps the key space small enough to eyeball."""
|
||||
m = _fresh(monkeypatch)
|
||||
lines = []
|
||||
monkeypatch.setattr(m, "_upsert", lambda *a: None)
|
||||
monkeypatch.setattr(m.audit, "log_activity", lambda tag, rec: lines.append(rec))
|
||||
m.record("share", {"view": "home", "prop": "link"},
|
||||
referer="https://reddit.com/r/weather", host="thermograph.org")
|
||||
assert lines[0]["referrer"] == ""
|
||||
|
||||
|
||||
def test_record_never_raises(monkeypatch):
|
||||
m = _fresh(monkeypatch)
|
||||
|
||||
def boom(*a, **k):
|
||||
raise RuntimeError("sink down")
|
||||
|
||||
monkeypatch.setattr(m, "_upsert", boom)
|
||||
monkeypatch.setattr(m.audit, "log_activity", boom)
|
||||
m.record("view.open", {"view": "home"}) # must not propagate
|
||||
|
||||
|
||||
# --- Tier B: the ephemeral session id ------------------------------------------
|
||||
# The session field is the one place a client could put arbitrary text into
|
||||
# storage, so it is validated by SHAPE and dropped whole on any mismatch. These
|
||||
# are the tests that keep it from quietly becoming a free-text column.
|
||||
|
||||
def test_session_id_accepts_only_the_exact_minted_shape(monkeypatch):
|
||||
m = _fresh(monkeypatch)
|
||||
good = "AbCd1234_-efGH5678ijK" # 21 chars: too short
|
||||
assert m.validate_session(good) == ""
|
||||
assert m.validate_session("A" * 22) == "A" * 22
|
||||
assert m.validate_session("aZ0_-" + "b" * 17) == "aZ0_-" + "b" * 17
|
||||
|
||||
|
||||
def test_session_id_rejects_anything_that_could_carry_text(monkeypatch):
|
||||
m = _fresh(monkeypatch)
|
||||
for bad in ("user@example.com", "47.6062,-122.3321", "a" * 200, "", None, 12345,
|
||||
"AAAAAAAAAAAAAAAAAAAA<>", "has space here aaaaaaa", {"id": "x"},
|
||||
"user-id-42", "AAAAAAAAAAAAAAAAAAAAA;DROP"):
|
||||
assert m.validate_session(bad) == "", bad
|
||||
|
||||
|
||||
def test_sequence_number_is_bounded(monkeypatch):
|
||||
m = _fresh(monkeypatch)
|
||||
assert m.validate_seq(0) == 0
|
||||
assert m.validate_seq(m.MAX_SEQ) == m.MAX_SEQ
|
||||
for bad in (-1, m.MAX_SEQ + 1, 10**9, "3", 3.0, True, None):
|
||||
assert m.validate_seq(bad) == -1, bad
|
||||
|
||||
|
||||
def test_session_rows_need_their_own_flag(monkeypatch):
|
||||
"""Tier A must record with Tier B off — the anonymous signal never depends on
|
||||
consent rates or on the session tier being enabled."""
|
||||
m = _fresh(monkeypatch)
|
||||
monkeypatch.setattr(m, "SESSIONS_ENABLED", False)
|
||||
agg, rows = [], []
|
||||
monkeypatch.setattr(m, "_upsert", lambda *a: agg.append(a))
|
||||
monkeypatch.setattr(m, "_insert_session", lambda *a: rows.append(a))
|
||||
monkeypatch.setattr(m.audit, "log_activity", lambda *a: None)
|
||||
m.record("view.open", {"view": "home"}, session="A" * 22, seq=1)
|
||||
assert len(agg) == 1 and rows == []
|
||||
|
||||
|
||||
def test_session_row_written_when_consented_and_enabled(monkeypatch):
|
||||
m = _fresh(monkeypatch)
|
||||
monkeypatch.setattr(m, "SESSIONS_ENABLED", True)
|
||||
agg, rows = [], []
|
||||
monkeypatch.setattr(m, "_upsert", lambda *a: agg.append(a))
|
||||
monkeypatch.setattr(m, "_insert_session", lambda *a: rows.append(a))
|
||||
monkeypatch.setattr(m.audit, "log_activity", lambda *a: None)
|
||||
m.record("place.pick", {"view": "home", "prop": "map"}, session="B" * 22, seq=4)
|
||||
assert len(agg) == 1
|
||||
(sid, seq, event, dims, _ref), = rows
|
||||
assert (sid, seq, event) == ("B" * 22, 4, "place.pick")
|
||||
assert dims["prop"] == "map"
|
||||
|
||||
|
||||
def test_bad_session_id_does_not_cost_the_aggregate(monkeypatch):
|
||||
m = _fresh(monkeypatch)
|
||||
monkeypatch.setattr(m, "SESSIONS_ENABLED", True)
|
||||
agg, rows = [], []
|
||||
monkeypatch.setattr(m, "_upsert", lambda *a: agg.append(a))
|
||||
monkeypatch.setattr(m, "_insert_session", lambda *a: rows.append(a))
|
||||
monkeypatch.setattr(m.audit, "log_activity", lambda *a: None)
|
||||
m.record("view.open", {"view": "home"}, session="not-a-valid-id", seq=1)
|
||||
assert len(agg) == 1 and rows == []
|
||||
|
||||
|
||||
def test_session_id_never_reaches_the_log_stream(monkeypatch):
|
||||
"""The activity JSONL goes to Loki with its own retention and access rules;
|
||||
a behavioural sequence must not be duplicated into it."""
|
||||
m = _fresh(monkeypatch)
|
||||
monkeypatch.setattr(m, "SESSIONS_ENABLED", True)
|
||||
lines = []
|
||||
monkeypatch.setattr(m, "_upsert", lambda *a: None)
|
||||
monkeypatch.setattr(m, "_insert_session", lambda *a: None)
|
||||
monkeypatch.setattr(m.audit, "log_activity", lambda tag, rec: lines.append(rec))
|
||||
m.record("view.open", {"view": "home"}, session="C" * 22, seq=2)
|
||||
assert "C" * 22 not in str(lines)
|
||||
assert "session" not in lines[0] and "seq" not in lines[0]
|
||||
|
||||
|
||||
# --- the one free-text field: zero-result search queries ------------------------
|
||||
|
||||
def test_search_miss_normalisation(monkeypatch):
|
||||
m = _fresh(monkeypatch)
|
||||
assert m.normalize_search_miss(" Saint-Étienne ") == "saint-étienne"
|
||||
assert m.normalize_search_miss("NEW\tYORK\n CITY") == "new york city"
|
||||
assert m.normalize_search_miss("Königsberg") == m.normalize_search_miss("KÖNIGSBERG")
|
||||
# A postcode-length digit run is a legitimate place query and survives.
|
||||
assert m.normalize_search_miss("98101") == "98101"
|
||||
|
||||
|
||||
def test_search_miss_rejects_anything_with_a_personal_data_smell(monkeypatch):
|
||||
m = _fresh(monkeypatch)
|
||||
for bad in (
|
||||
"someone@example.com", # email
|
||||
"https://example.com/x", # URL
|
||||
"www.example.com", # hostname
|
||||
"call me on 07700900123", # long digit run
|
||||
"x" * 60, # too long
|
||||
"my house is the third one on the left", # a sentence
|
||||
"$$$ ???", # punctuation soup
|
||||
"47.6062,-122.3321", # coordinates
|
||||
None, 42, "",
|
||||
):
|
||||
assert m.normalize_search_miss(bad) == "", bad
|
||||
|
||||
|
||||
def test_search_miss_needs_its_own_flag(monkeypatch):
|
||||
m = _fresh(monkeypatch)
|
||||
monkeypatch.setattr(m, "SEARCH_MISS_ENABLED", False)
|
||||
seen = []
|
||||
monkeypatch.setattr(m, "_upsert_search_miss", lambda q: seen.append(q))
|
||||
m.record_search_miss("atlantis")
|
||||
assert seen == []
|
||||
monkeypatch.setattr(m, "SEARCH_MISS_ENABLED", True)
|
||||
m.record_search_miss("atlantis")
|
||||
m.record_search_miss("someone@example.com")
|
||||
assert seen == ["atlantis"]
|
||||
|
||||
|
||||
def test_search_miss_prune_enforces_a_k_anonymity_floor(monkeypatch):
|
||||
"""The strongest mitigation on the free-text field: a string only one or two
|
||||
people typed is deleted; one several people typed is a product signal."""
|
||||
m = _fresh(monkeypatch)
|
||||
assert m.SEARCH_MISS_MIN_COUNT >= 3
|
||||
assert f"count < {m.SEARCH_MISS_MIN_COUNT}" in m.PRUNE_SEARCH_MISS_SQL
|
||||
assert f"{m.SEARCH_MISS_PRUNE_DAYS} days" in m.PRUNE_SEARCH_MISS_SQL
|
||||
|
|
@ -322,3 +322,38 @@ def test_score_shape_and_conditional_revalidation(score_client):
|
|||
# Second GET replays the exact same bytes from the derived store.
|
||||
r2 = score_client.get("/thermograph/api/v2/score", params=Q)
|
||||
assert r2.status_code == 200 and r2.content == r.content
|
||||
|
||||
|
||||
# --- failed-search capture (server-side only) ----------------------------------
|
||||
|
||||
def test_zero_result_search_records_the_query_text_server_side(client, monkeypatch):
|
||||
"""Decision 2: the miss text is captured in the handler, never sent from the
|
||||
browser. Only misses — a query that found something is never stored."""
|
||||
from core import events
|
||||
seen = []
|
||||
monkeypatch.setattr(events, "ENABLED", True)
|
||||
monkeypatch.setattr(events, "SEARCH_MISS_ENABLED", True)
|
||||
monkeypatch.setattr(events, "_upsert", lambda *a: None)
|
||||
monkeypatch.setattr(events, "_upsert_search_miss", lambda q: seen.append(q))
|
||||
monkeypatch.setattr(events.audit, "log_activity", lambda *a: None)
|
||||
monkeypatch.setattr(places, "suggest", lambda q, up, lim: ([], None))
|
||||
client.get("/thermograph/api/v2/suggest", params={"q": "Atlantis"})
|
||||
assert seen == ["atlantis"]
|
||||
|
||||
monkeypatch.setattr(places, "suggest",
|
||||
lambda q, up, lim: ([{"name": "Seattle"}], None))
|
||||
client.get("/thermograph/api/v2/suggest", params={"q": "Seattle"})
|
||||
assert seen == ["atlantis"] # a hit is never stored as text
|
||||
|
||||
|
||||
def test_search_miss_text_is_not_written_to_the_access_log(client, monkeypatch):
|
||||
"""The app's own access log records url.path, never the query string — so the
|
||||
search text cannot leak into it. (Caddy's log is a separate concern: it logs
|
||||
the full URI, which is why the miss capture must stay server-side and why the
|
||||
Caddy log format needs the query stripped -- see UI-EVENTS.md.)"""
|
||||
from core import audit
|
||||
lines = []
|
||||
monkeypatch.setattr(audit, "log_access", lambda rec: lines.append(rec))
|
||||
monkeypatch.setattr(places, "suggest", lambda q, up, lim: ([], None))
|
||||
client.get("/thermograph/api/v2/suggest", params={"q": "somewhere-secret"})
|
||||
assert lines and all("somewhere-secret" not in str(v) for rec in lines for v in rec.values())
|
||||
|
|
|
|||
194
backend/tests/web/test_event_beacon.py
Normal file
194
backend/tests/web/test_event_beacon.py
Normal file
|
|
@ -0,0 +1,194 @@
|
|||
"""Route-level tests for the public product-event beacon (POST /api/v2/event).
|
||||
|
||||
The endpoint is unauthenticated and reachable by anything on the internet — on a
|
||||
site where ~60% of inbound traffic is crawlers and scanners — so these tests are
|
||||
mostly about what it REFUSES to do: grow storage, echo anything back, or accept
|
||||
a payload big enough to be worth abusing.
|
||||
"""
|
||||
import json
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from core import events
|
||||
from web import app as appmod
|
||||
|
||||
EVENT_URL = "/thermograph/api/v2/event"
|
||||
# Browsers set this on fetch/sendBeacon; a bare scanner POST does not.
|
||||
FIRST_PARTY = {"sec-fetch-site": "same-origin"}
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _fresh_rate_limiter():
|
||||
"""TestClient always presents the same client IP, so without this one test's
|
||||
beacon traffic silently rate-limits the next one's."""
|
||||
from core import metrics
|
||||
metrics.reset_rate_limiter()
|
||||
yield
|
||||
metrics.reset_rate_limiter()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
return TestClient(appmod.app)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def recorded(monkeypatch):
|
||||
"""Capture what reaches the recorder, with the flag forced on."""
|
||||
seen = []
|
||||
monkeypatch.setattr(events, "ENABLED", True)
|
||||
monkeypatch.setattr(events, "_upsert", lambda *a: None)
|
||||
monkeypatch.setattr(events.audit, "log_activity", lambda tag, rec: seen.append(rec))
|
||||
return seen
|
||||
|
||||
|
||||
def test_beacon_always_answers_204_and_returns_no_body(client, recorded):
|
||||
"""Counted, rejected, or rate-limited must be indistinguishable to a prober."""
|
||||
for body in ({"event": "view.open", "props": {"view": "home"}},
|
||||
{"event": "not-a-real-event"},
|
||||
{"nonsense": True},
|
||||
[1, 2, 3],
|
||||
"plain string"):
|
||||
r = client.post(EVENT_URL, content=json.dumps(body), headers=FIRST_PARTY)
|
||||
assert r.status_code == 204
|
||||
assert r.content == b""
|
||||
|
||||
|
||||
def test_valid_event_is_recorded_with_allowlisted_dimensions(client, recorded):
|
||||
client.post(EVENT_URL, json={"event": "view.control",
|
||||
"props": {"view": "compare", "prop": "unit", "value": "C"}},
|
||||
headers=FIRST_PARTY)
|
||||
assert recorded == [{"event": "view.control", "view": "compare", "prop": "unit",
|
||||
"value": "C", "referrer": ""}]
|
||||
|
||||
|
||||
def test_batched_payload_is_accepted_and_capped(client, recorded):
|
||||
items = [{"event": "share", "props": {"prop": "link"}}] * (events.MAX_BATCH + 5)
|
||||
client.post(EVENT_URL, json={"events": items}, headers=FIRST_PARTY)
|
||||
assert len(recorded) == events.MAX_BATCH
|
||||
|
||||
|
||||
def test_a_full_batch_of_real_events_fits_under_the_byte_cap(client, recorded):
|
||||
"""Regression: if MAX_BODY_BYTES ever drops below a full MAX_BATCH payload,
|
||||
the byte cap silently eats legitimate batches instead of abusive ones."""
|
||||
items = [{"event": "view.control",
|
||||
"props": {"view": "compare", "prop": "chart_metric", "value": "precip"}}
|
||||
] * events.MAX_BATCH
|
||||
assert len(json.dumps({"events": items})) < events.MAX_BODY_BYTES
|
||||
client.post(EVENT_URL, json={"events": items}, headers=FIRST_PARTY)
|
||||
assert len(recorded) == events.MAX_BATCH
|
||||
|
||||
|
||||
def test_oversized_body_is_dropped_without_being_parsed(client, recorded):
|
||||
"""The endpoint must not become storage: a body past the cap is never read
|
||||
into a record at all."""
|
||||
fat = {"event": "view.open", "props": {"view": "home"},
|
||||
"junk": "x" * (events.MAX_BODY_BYTES * 4)}
|
||||
r = client.post(EVENT_URL, content=json.dumps(fat), headers=FIRST_PARTY)
|
||||
assert r.status_code == 204
|
||||
assert recorded == []
|
||||
|
||||
|
||||
def test_free_text_and_coordinates_never_survive(client, recorded):
|
||||
"""Anything a client puts in a dimension slot is allowlisted or bucketed —
|
||||
a leaked query string or lat/lon cannot reach a sink."""
|
||||
client.post(EVENT_URL, json={"event": "place.pick",
|
||||
"props": {"view": "home", "prop": "47.6062,-122.3321"}},
|
||||
headers=FIRST_PARTY)
|
||||
client.post(EVENT_URL, json={"event": "view.control",
|
||||
"props": {"view": "home", "prop": "chart_metric",
|
||||
"value": "user@example.com"}},
|
||||
headers=FIRST_PARTY)
|
||||
stored = {v for rec in recorded for v in rec.values()}
|
||||
assert not any("47.6" in s or "@" in s for s in stored)
|
||||
|
||||
|
||||
def test_cross_site_post_is_dropped(client, recorded):
|
||||
r = client.post(EVENT_URL, json={"event": "view.open", "props": {"view": "home"}},
|
||||
headers={"sec-fetch-site": "cross-site"})
|
||||
assert r.status_code == 204
|
||||
assert recorded == []
|
||||
|
||||
|
||||
def test_flag_off_records_nothing(client, monkeypatch):
|
||||
"""The whole feature ships inert: with THERMOGRAPH_EVENTS unset the beacon
|
||||
still answers, and nothing reaches the durable sink."""
|
||||
seen = []
|
||||
monkeypatch.setattr(events, "ENABLED", False)
|
||||
monkeypatch.setattr(events, "_upsert", lambda *a: seen.append(a))
|
||||
monkeypatch.setattr(events.audit, "log_activity", lambda tag, rec: seen.append(rec))
|
||||
r = client.post(EVENT_URL, json={"event": "view.open", "props": {"view": "home"}},
|
||||
headers=FIRST_PARTY)
|
||||
assert r.status_code == 204
|
||||
assert seen == []
|
||||
|
||||
|
||||
def test_legacy_single_event_shape_still_counts(client, monkeypatch):
|
||||
"""The four home.* aggregate counters already in production keep working —
|
||||
they are unflagged and predate the schema."""
|
||||
from core import metrics
|
||||
seen = []
|
||||
monkeypatch.setattr(metrics, "_store", type("S", (), {
|
||||
"record_event": lambda self, *a: seen.append(a)})())
|
||||
client.post(EVENT_URL, json={"event": "home.locate"}, headers=FIRST_PARTY)
|
||||
assert seen and seen[0][0] == "home.locate"
|
||||
|
||||
|
||||
def test_beacon_is_not_counted_as_inbound_traffic(client):
|
||||
"""It reports traffic; counting it would double every interaction."""
|
||||
from core import metrics
|
||||
assert metrics.classify_inbound(EVENT_URL, "/thermograph") == "event"
|
||||
|
||||
|
||||
# --- Tier B over the wire -------------------------------------------------------
|
||||
|
||||
@pytest.fixture
|
||||
def session_rows(monkeypatch, recorded):
|
||||
rows = []
|
||||
monkeypatch.setattr(events, "SESSIONS_ENABLED", True)
|
||||
monkeypatch.setattr(events, "_insert_session", lambda *a: rows.append(a))
|
||||
return rows
|
||||
|
||||
|
||||
def test_session_and_seq_travel_with_a_batch(client, session_rows):
|
||||
client.post(EVENT_URL, json={
|
||||
"session": "D" * 22,
|
||||
"events": [{"event": "view.open", "props": {"view": "home"}, "seq": 1},
|
||||
{"event": "place.pick", "props": {"view": "home", "prop": "map"}, "seq": 2}],
|
||||
}, headers=FIRST_PARTY)
|
||||
assert [(r[0], r[1], r[2]) for r in session_rows] == [
|
||||
("D" * 22, 1, "view.open"), ("D" * 22, 2, "place.pick")]
|
||||
|
||||
|
||||
def test_a_forged_session_field_cannot_smuggle_text(client, session_rows, recorded):
|
||||
"""The session field is the only client-controlled string that reaches a
|
||||
sink, so a malformed one must lose its payload entirely — while the
|
||||
anonymous aggregate still records."""
|
||||
for forged in ("'; DROP TABLE ui_event_session; --", "user@example.com",
|
||||
"47.6062,-122.3321", "x" * 500):
|
||||
client.post(EVENT_URL, json={"session": forged, "seq": 1,
|
||||
"event": "view.open", "props": {"view": "home"}},
|
||||
headers=FIRST_PARTY)
|
||||
assert session_rows == []
|
||||
assert len(recorded) == 4 # Tier A recorded every one of them
|
||||
|
||||
|
||||
def test_sessions_flag_off_records_only_the_aggregate(client, monkeypatch, recorded):
|
||||
rows = []
|
||||
monkeypatch.setattr(events, "SESSIONS_ENABLED", False)
|
||||
monkeypatch.setattr(events, "_insert_session", lambda *a: rows.append(a))
|
||||
client.post(EVENT_URL, json={"session": "E" * 22, "seq": 1,
|
||||
"event": "view.open", "props": {"view": "home"}},
|
||||
headers=FIRST_PARTY)
|
||||
assert rows == [] and len(recorded) == 1
|
||||
|
||||
|
||||
def test_beacon_never_reads_the_account_session_cookie(client, session_rows, recorded):
|
||||
"""Even a signed-in visitor's events stay unlinked to the account: the route
|
||||
does not consult the auth cookie, so there is nothing to join on."""
|
||||
client.cookies.set("thermographauth", "pretend-jwt-value")
|
||||
client.post(EVENT_URL, json={"event": "view.open", "props": {"view": "home"}},
|
||||
headers=FIRST_PARTY)
|
||||
assert "pretend-jwt" not in str(recorded) + str(session_rows)
|
||||
assert recorded and all("user" not in k for rec in recorded for k in rec)
|
||||
|
|
@ -20,6 +20,7 @@ from fastapi.responses import JSONResponse, RedirectResponse
|
|||
from accounts import api_accounts
|
||||
from api import content_routes
|
||||
from core import audit
|
||||
from core import events
|
||||
from data import climate
|
||||
from notifications import digest
|
||||
from notifications import discord_bot
|
||||
|
|
@ -452,6 +453,23 @@ def api_suggest(q: str = Query(..., min_length=1)):
|
|||
results, corrected = places.suggest(q, _no_upstream, _SUGGEST_LIMIT)
|
||||
except Exception as e: # noqa: BLE001 - nothing at all to serve
|
||||
raise HTTPException(status_code=502, detail=f"geocoding failed: {e}")
|
||||
# "Did search work?" is recorded HERE, not in the browser: the server already
|
||||
# knows the answer, every surface that searches is covered for free, and the
|
||||
# query string never has to leave this handler. Inert unless
|
||||
# THERMOGRAPH_EVENTS is on. NOTE: the type-ahead fires once per debounced
|
||||
# keystroke, so this counts *keystroke batches*, not distinct searches — read
|
||||
# the miss RATE, not the miss count.
|
||||
events.record("place.search",
|
||||
{"prop": "hit" if results and not corrected
|
||||
else "typo" if results else "miss"})
|
||||
if not results:
|
||||
# The one free-text capture in the system, and server-side only: what
|
||||
# people looked for and did NOT find names the gaps in the place index.
|
||||
# events.record_search_miss owns every mitigation — it rejects anything
|
||||
# with a personal-data smell outright, stores a per-day count rather than
|
||||
# a row per search, and a daily prune deletes strings only one or two
|
||||
# people ever typed. Behind its own flag, off by default.
|
||||
events.record_search_miss(q)
|
||||
return {"results": results, "corrected": corrected}
|
||||
|
||||
|
||||
|
|
@ -779,36 +797,123 @@ def api_metrics(request: Request):
|
|||
return snap
|
||||
|
||||
|
||||
async def api_event(request: Request) -> Response:
|
||||
"""Record one product event (a tap on "use my location", a digest signup, …).
|
||||
def _same_origin(request: Request) -> bool:
|
||||
"""A soft first-party check for the public beacon.
|
||||
|
||||
Deliberately minimal: the body carries only an allowlisted event name. The
|
||||
referrer is read from this request's own Referer header — a client-supplied
|
||||
one would be forgeable and buys nothing — and is reduced to a bare domain.
|
||||
No cookies, no per-visitor id, no full URLs.
|
||||
|
||||
Always answers 204, whether the event was counted, rejected by the allowlist,
|
||||
or dropped by the rate limiter, so probing the endpoint reveals nothing. That
|
||||
also suits navigator.sendBeacon, which ignores the response.
|
||||
Every browser that can run our JS sets ``Sec-Fetch-Site`` on a fetch or
|
||||
sendBeacon; a curl/scanner POST does not, and a cross-site forgery reports
|
||||
``cross-site``. Neither header is a security boundary (both are trivially
|
||||
spoofable by anything that bothers), but together they filter out the naive
|
||||
majority — which is exactly what "don't let the analytics table fill with
|
||||
junk" needs. Anything suspicious is dropped silently; the response is 204
|
||||
either way.
|
||||
"""
|
||||
name = ""
|
||||
site = request.headers.get("sec-fetch-site")
|
||||
if site is not None:
|
||||
return site in ("same-origin", "same-site", "none")
|
||||
origin = request.headers.get("origin")
|
||||
if origin:
|
||||
host = (request.headers.get("host") or "").partition(":")[0].lower()
|
||||
return host != "" and origin.rpartition("/")[2].partition(":")[0].lower().removeprefix("www.") \
|
||||
== host.removeprefix("www.")
|
||||
# No Sec-Fetch-Site and no Origin: an old browser, or a bot. Keep it — the
|
||||
# allowlist + rate limiter already bound the damage — but it is the bucket
|
||||
# to look at first if the numbers ever look wrong.
|
||||
return True
|
||||
|
||||
|
||||
async def _read_capped(request: Request, limit: int) -> bytes:
|
||||
"""At most ``limit`` bytes of the request body. A real beacon is ~60 bytes,
|
||||
so anything larger is either a bug or an attempt to use the endpoint as
|
||||
storage: stop reading rather than buffering it."""
|
||||
declared = request.headers.get("content-length")
|
||||
if declared and declared.isdigit() and int(declared) > limit:
|
||||
return b""
|
||||
chunks: "list[bytes]" = []
|
||||
size = 0
|
||||
async for chunk in request.stream():
|
||||
size += len(chunk)
|
||||
if size > limit:
|
||||
return b""
|
||||
chunks.append(chunk)
|
||||
return b"".join(chunks)
|
||||
|
||||
|
||||
async def api_event(request: Request) -> Response:
|
||||
"""Record one or more product events (a tap on "use my location", a chart
|
||||
metric switch, a dead end the visitor hit, …).
|
||||
|
||||
Deliberately minimal: the body carries an allowlisted event name and a few
|
||||
allowlisted enum properties — no free text, no coordinates, and **the account
|
||||
session cookie is never read on this route**, so nothing recorded here can be
|
||||
tied to an identified person even when the visitor is signed in. The referrer
|
||||
is taken from this request's own Referer header (a client-supplied one would
|
||||
be forgeable and buys nothing) and reduced to a bare domain.
|
||||
|
||||
Body shapes accepted::
|
||||
|
||||
{"event": "view.open", "props": {"view": "calendar", "prop": "nav"}}
|
||||
{"events": [{"event": ..., "props": {...}}, ...]} # batched, <= MAX_BATCH
|
||||
|
||||
A *consented* client additionally sends ``session`` — an ephemeral, per-tab,
|
||||
self-rotating id minted in the browser (see frontend/static/consent.js and
|
||||
track.js) — plus a per-event ``seq``. Both are validated by shape and dropped
|
||||
whole if they are not exactly what is expected: ``session`` is the one field
|
||||
a client could otherwise use to smuggle free text into storage. The anonymous
|
||||
aggregate records identically with or without them, so a refused id costs the
|
||||
primary signal nothing.
|
||||
|
||||
Always answers 204 — counted, rejected by the allowlist, over the body cap,
|
||||
or dropped by the rate limiter all look identical, so probing the endpoint
|
||||
reveals nothing. That also suits navigator.sendBeacon, which ignores the
|
||||
response entirely.
|
||||
"""
|
||||
raw = await _read_capped(request, events.MAX_BODY_BYTES)
|
||||
if not raw or not _same_origin(request):
|
||||
return Response(status_code=204)
|
||||
try:
|
||||
body = await request.json()
|
||||
if isinstance(body, dict):
|
||||
name = str(body.get("event") or "")
|
||||
except Exception: # noqa: BLE001 - a junk body is just a no-op event
|
||||
pass
|
||||
if name:
|
||||
await run_in_threadpool(
|
||||
metrics.record_event,
|
||||
name,
|
||||
referer=request.headers.get("referer"),
|
||||
host=request.headers.get("host"),
|
||||
ip=_client_ip(request),
|
||||
)
|
||||
body = json.loads(raw)
|
||||
except Exception: # noqa: BLE001 - a junk body is just a no-op
|
||||
return Response(status_code=204)
|
||||
if not isinstance(body, dict):
|
||||
return Response(status_code=204)
|
||||
|
||||
batch = body.get("events")
|
||||
if not isinstance(batch, list):
|
||||
batch = [body]
|
||||
batch = [e for e in batch[:events.MAX_BATCH] if isinstance(e, dict)]
|
||||
|
||||
referer = request.headers.get("referer")
|
||||
host = request.headers.get("host")
|
||||
# The IP is a rate-limit key for the current minute and nothing else: it is
|
||||
# never written to the counters, the aggregate table, or the activity log.
|
||||
ip = _client_ip(request)
|
||||
# One session id for the whole batch (a batch is by construction one tab's
|
||||
# events), but each event carries its own sequence number.
|
||||
session = body.get("session")
|
||||
for item in batch:
|
||||
name = str(item.get("event") or "")
|
||||
if not name:
|
||||
continue
|
||||
props = item.get("props")
|
||||
await run_in_threadpool(_record_event, name, props, referer, host, ip,
|
||||
session, item.get("seq"))
|
||||
return Response(status_code=204)
|
||||
|
||||
|
||||
def _record_event(name, props, referer, host, ip, session=None, seq=None) -> None:
|
||||
"""Fan one event out to every sink (blocking; called via the threadpool).
|
||||
|
||||
``metrics`` keeps the since-start live counters the ops endpoint already
|
||||
serves — untouched, and still the rate-limit gate. ``events`` adds the durable
|
||||
anonymous hourly aggregate plus, for a consented client only, the
|
||||
session-scoped row. Both are inert unless their flags are on."""
|
||||
if not metrics.rate_ok(ip):
|
||||
return
|
||||
metrics.record_event(name, referer=referer, host=host, rate_limit=False)
|
||||
events.record(name, props, referer=referer, host=host, session=session, seq=seq)
|
||||
|
||||
|
||||
# --- API versioning --------------------------------------------------------
|
||||
# Non-backward-compatible additions land in a new version; every version stays
|
||||
# mounted and served simultaneously. v1 is the original contract (grade/geocode);
|
||||
|
|
|
|||
|
|
@ -96,6 +96,12 @@ def _page(name):
|
|||
verify = content.head_verify_html()
|
||||
if verify:
|
||||
html = html.replace("<head>", f"<head>\n {verify}", 1)
|
||||
# Same flag stamp the SSR base template applies: track.js sends
|
||||
# nothing unless it finds it. Folded into the memoized template
|
||||
# because it is constant for the process's life.
|
||||
if content.EVENTS_ENABLED:
|
||||
html = html.replace('<html lang="en">',
|
||||
'<html lang="en" data-tg-events="1">', 1)
|
||||
_template.append(html)
|
||||
return _template[0]
|
||||
|
||||
|
|
|
|||
|
|
@ -36,6 +36,14 @@ BASE = f"/{_BASE}" if _BASE else ""
|
|||
# references stay relative (ASSET_BASE == BASE). Repo-split Stage 5.
|
||||
ASSET_BASE = os.environ.get("THERMOGRAPH_API_BASE_PUBLIC", "").rstrip("/") or BASE
|
||||
|
||||
# UI product-event instrumentation (backend/core/events.py). OFF unless this is
|
||||
# set, and set per-environment -- the flag lives in BOTH services because each
|
||||
# renders HTML: this one stamps data-tg-events on <html>, which is the only
|
||||
# thing that makes track.js send anything, and backend's copy is what decides
|
||||
# whether the beacon records what arrives. Both must be on for events to flow.
|
||||
EVENTS_ENABLED = os.environ.get("THERMOGRAPH_EVENTS", "").strip().lower() in (
|
||||
"1", "true", "yes", "on")
|
||||
|
||||
_env = Environment(
|
||||
loader=FileSystemLoader(paths.TEMPLATES_DIR),
|
||||
autoescape=select_autoescape(["html", "xml", "j2"]),
|
||||
|
|
@ -76,6 +84,7 @@ def _respond_html(request: Request, template: str, **ctx) -> Response:
|
|||
# origin recovers exactly today's behavior in the default topology.
|
||||
asset_base_url = ASSET_BASE if ASSET_BASE.startswith(("http://", "https://")) else f"{o}{ASSET_BASE}"
|
||||
ctx.setdefault("unit_default", fmt.current_unit())
|
||||
ctx.setdefault("events_enabled", EVENTS_ENABLED)
|
||||
html = _env.get_template(template).render(
|
||||
base=BASE, asset_base=ASSET_BASE, asset_base_url=asset_base_url,
|
||||
origin=o, base_url=f"{o}{BASE}", **ctx)
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ import { loadView, TTL, prefetchViews } from "./cache.js";
|
|||
import { initFindButton, setFindLabel } from "./mappicker.js";
|
||||
import { W, PW, H, PL, PR, plotTop, plotBot, xLabY, setChartWidth, chartPalette,
|
||||
tempChart, precipChart, dryChart, attachChartHover } from "./chart.js";
|
||||
import { track } from "./digest.js";
|
||||
import { track, trackLegacy, currentView } from "./track.js";
|
||||
import { TIER_COLORS, SCALE_TEMP, drynessColor, pctOrd, esc, todayISO, placeLabel,
|
||||
tierKeySegs, GUIDE_LINK, fmtPrecip, fmtWind, fmtHumid } from "./shared.js";
|
||||
|
||||
|
|
@ -46,7 +46,10 @@ function selectLocation(lat, lon) {
|
|||
const findBtn = document.getElementById("find-btn");
|
||||
const locLabel = document.getElementById("loc-label");
|
||||
initFindButton(findBtn, "Find a location",
|
||||
() => selected, (lat, lon) => selectLocation(lat, lon));
|
||||
() => selected, (lat, lon, method) => {
|
||||
track("place.pick", { view: "home", prop: method || "map" });
|
||||
selectLocation(lat, lon);
|
||||
});
|
||||
|
||||
// ---- hero ----
|
||||
// The hero's grade card is server-rendered showing the most unusual city we're
|
||||
|
|
@ -78,7 +81,7 @@ if (heroLocate && !canLocate) {
|
|||
}
|
||||
|
||||
heroLocate?.addEventListener("click", () => {
|
||||
track("home.locate");
|
||||
trackLegacy("home.locate");
|
||||
const label = heroLocate.textContent;
|
||||
const restore = () => {
|
||||
heroLocate.disabled = false;
|
||||
|
|
@ -93,10 +96,16 @@ heroLocate?.addEventListener("click", () => {
|
|||
navigator.geolocation.getCurrentPosition(
|
||||
(pos) => {
|
||||
restore();
|
||||
// Records only THAT a location was obtained and by which route — never
|
||||
// the coordinates, which are location data about a person.
|
||||
track("place.pick", { view: "home", prop: "geolocate" });
|
||||
selectLocation(pos.coords.latitude, pos.coords.longitude);
|
||||
},
|
||||
(err) => {
|
||||
restore();
|
||||
// A dead end the visitor actually hit: the button looked like it worked
|
||||
// and then nothing happened. Worth a counter of its own.
|
||||
track("deadend", { view: "home", prop: "geo_denied" });
|
||||
// Always say something: a declined prompt or a timeout must not look
|
||||
// like the button did nothing.
|
||||
locateMsg(
|
||||
|
|
@ -160,7 +169,10 @@ async function runGrade() {
|
|||
url, ttl: TTL.grade, seq: ++gradeSeq, current: () => gradeSeq,
|
||||
spinner: () => { results.innerHTML = `<p class="spinner">Fetching decades of climate history…</p>`; },
|
||||
onData: render,
|
||||
onError: (err) => { results.innerHTML = `<p class="error">${err.message}</p>`; },
|
||||
onError: (err) => {
|
||||
results.innerHTML = `<p class="error">${err.message}</p>`;
|
||||
track("deadend", { view: "home", prop: navigator.onLine === false ? "offline" : "grade_error" });
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
|
|
@ -505,6 +517,7 @@ function buildChart(data) {
|
|||
const b = e.target.closest("button[data-metric]");
|
||||
if (!b || b.dataset.metric === chartMetric) return;
|
||||
chartMetric = b.dataset.metric;
|
||||
track("view.control", { view: "home", prop: "chart_metric", value: chartMetric });
|
||||
try { localStorage.setItem("thermograph:chartMetric", chartMetric); } catch (e) {}
|
||||
buildChart(data);
|
||||
});
|
||||
|
|
@ -514,6 +527,7 @@ function buildChart(data) {
|
|||
// ---- share ----
|
||||
function copyShareLink() {
|
||||
if (!selected) return;
|
||||
track("share", { view: "home", prop: "link" });
|
||||
const url = `${location.origin}${location.pathname}${locHash(selected.lat, selected.lon, dateInput.value)}`;
|
||||
navigator.clipboard.writeText(url).then(
|
||||
() => flashBtn("btn-link", "✓ Copied!"),
|
||||
|
|
@ -590,6 +604,7 @@ function downloadChartPng() {
|
|||
a.click();
|
||||
setTimeout(() => URL.revokeObjectURL(a.href), 1000);
|
||||
flashBtn("btn-png", "✓ Saved");
|
||||
track("share", { view: "home", prop: "png" });
|
||||
}, "image/png");
|
||||
};
|
||||
img.src = url;
|
||||
|
|
@ -619,3 +634,11 @@ function restoreFromHash() {
|
|||
selectLocation(loc.lat, loc.lon);
|
||||
}
|
||||
restoreFromHash();
|
||||
|
||||
// One view.open per page load. `prop` is how the view was reached: an explicit
|
||||
// link (a shared permalink carries a location hash), a remembered place, or a
|
||||
// cold arrival. This is the denominator every other event is read against.
|
||||
track("view.open", {
|
||||
view: currentView(),
|
||||
prop: location.hash ? "link" : (selected ? "restore" : "direct"),
|
||||
});
|
||||
|
|
|
|||
|
|
@ -121,6 +121,10 @@
|
|||
</main>
|
||||
|
||||
<script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script>
|
||||
<!-- Product-event beacon. Inert unless the server flagged this page
|
||||
on the root element; see frontend/static/track.js. -->
|
||||
<script type="module" src="consent.js"></script>
|
||||
<script type="module" src="track.js"></script>
|
||||
<script type="module" src="calendar.js"></script>
|
||||
<script type="module" src="ios-install.js"></script>
|
||||
</body>
|
||||
|
|
|
|||
|
|
@ -177,6 +177,10 @@
|
|||
</main>
|
||||
|
||||
<script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script>
|
||||
<!-- Product-event beacon. Inert unless the server flagged this page
|
||||
on the root element; see frontend/static/track.js. -->
|
||||
<script type="module" src="consent.js"></script>
|
||||
<script type="module" src="track.js"></script>
|
||||
<script type="module" src="compare.js"></script>
|
||||
<script type="module" src="ios-install.js"></script>
|
||||
</body>
|
||||
|
|
|
|||
145
frontend/static/consent.js
Normal file
145
frontend/static/consent.js
Normal file
|
|
@ -0,0 +1,145 @@
|
|||
// Analytics consent: the opt-in gate in front of the session-scoped tier.
|
||||
//
|
||||
// WHY THIS EXISTS. The anonymous tier (hourly counts, no identifier, nothing
|
||||
// written to your device) needs no permission. The session tier does: it mints
|
||||
// an ephemeral id and stores it in sessionStorage, and storing an identifier on
|
||||
// someone's device engages ePrivacy Art. 5(3). Product analytics is not
|
||||
// "strictly necessary to provide a service the user explicitly requested", so
|
||||
// the exemption does not apply and the lawful basis is opt-in consent.
|
||||
//
|
||||
// RULES THIS FILE ENFORCES, all of them load-bearing:
|
||||
// * Nothing is stored and no id is minted before an explicit "Allow". Not on
|
||||
// first paint, not "pending a choice" — refusing by ignoring the banner must
|
||||
// leave exactly as little behind as refusing explicitly.
|
||||
// * Allow and No are the same size, same weight, same prominence. No pre-tick,
|
||||
// no "legitimate interest" second layer, no cookie-wall.
|
||||
// * Withdrawal is as easy as granting, reachable at any time from the footer
|
||||
// and the privacy page, and it deletes the live session id immediately.
|
||||
// * GPC / DNT are treated as a refusal already given: no banner is shown and
|
||||
// nothing is stored. Stricter than the letter of the law, and it means the
|
||||
// most privacy-conscious visitors are never nagged.
|
||||
//
|
||||
// The consent RECORD itself lives in localStorage. Storing a user's own choice
|
||||
// is strictly necessary to honour it, so it is exempt from the consent it
|
||||
// records — but it is deliberately the only thing kept across sessions.
|
||||
|
||||
const KEY = "thermograph:analytics-consent";
|
||||
//: Bump to re-ask everyone — only when what is collected materially changes.
|
||||
const POLICY_VERSION = 1;
|
||||
//: Re-ask a granter after this long. Never re-ask a refuser inside it: a banner
|
||||
//: that keeps reappearing after "no" is a dark pattern, not a question.
|
||||
const GRANT_TTL_MS = 183 * 24 * 3600 * 1000; // ~6 months
|
||||
const DENY_TTL_MS = 365 * 24 * 3600 * 1000; // ~12 months
|
||||
|
||||
export const signalledOptOut =
|
||||
navigator.globalPrivacyControl === true ||
|
||||
navigator.doNotTrack === "1" ||
|
||||
window.doNotTrack === "1";
|
||||
|
||||
const listeners = [];
|
||||
let state = null; // "granted" | "denied" | null (not asked)
|
||||
|
||||
function read() {
|
||||
try {
|
||||
const raw = localStorage.getItem(KEY);
|
||||
if (!raw) return null;
|
||||
const rec = JSON.parse(raw);
|
||||
if (rec.v !== POLICY_VERSION) return null;
|
||||
const ttl = rec.choice === "granted" ? GRANT_TTL_MS : DENY_TTL_MS;
|
||||
if (!rec.ts || Date.now() - rec.ts > ttl) return null;
|
||||
return rec.choice === "granted" ? "granted" : "denied";
|
||||
} catch { return null; }
|
||||
}
|
||||
|
||||
function write(choice) {
|
||||
try {
|
||||
localStorage.setItem(KEY, JSON.stringify({ v: POLICY_VERSION, choice, ts: Date.now() }));
|
||||
} catch { /* private mode: the choice just won't persist */ }
|
||||
}
|
||||
|
||||
/** "granted" | "denied" | null. GPC/DNT short-circuits to "denied". */
|
||||
export function consentState() {
|
||||
if (signalledOptOut) return "denied";
|
||||
if (state === null) state = read();
|
||||
return state;
|
||||
}
|
||||
|
||||
export const hasConsent = () => consentState() === "granted";
|
||||
|
||||
/** Called with the new state whenever it changes — track.js uses this to drop
|
||||
* the live session id the instant consent is withdrawn. */
|
||||
export function onConsentChange(cb) { listeners.push(cb); }
|
||||
|
||||
export function setConsent(choice) {
|
||||
state = choice === "granted" ? "granted" : "denied";
|
||||
write(state);
|
||||
hideBanner();
|
||||
paintControls();
|
||||
for (const cb of listeners) { try { cb(state); } catch { /* never break */ } }
|
||||
}
|
||||
|
||||
// --- the banner ---------------------------------------------------------------
|
||||
// Rendered only when instrumentation is enabled AND no choice has been recorded
|
||||
// AND no opt-out signal is present. It is a bar, not a modal: it never blocks
|
||||
// reading the page, which is the whole product.
|
||||
|
||||
let banner = null;
|
||||
|
||||
function hideBanner() {
|
||||
if (banner) { banner.remove(); banner = null; }
|
||||
}
|
||||
|
||||
function showBanner() {
|
||||
if (banner) return;
|
||||
banner = document.createElement("div");
|
||||
banner.className = "consent-bar";
|
||||
banner.setAttribute("role", "dialog");
|
||||
banner.setAttribute("aria-label", "Analytics");
|
||||
banner.innerHTML = `
|
||||
<p>Can we count how this page gets used? Anonymous, kept for 30 days, never
|
||||
shared, and never linked to you or to an account.
|
||||
<a class="consent-more" href="">What this collects</a></p>
|
||||
<div class="consent-actions">
|
||||
<button type="button" class="consent-no">No thanks</button>
|
||||
<button type="button" class="consent-yes">Allow</button>
|
||||
</div>`;
|
||||
// The privacy link is resolved against the page, not hardcoded, so it is right
|
||||
// under both the root-served and /thermograph-prefixed deployments.
|
||||
const more = banner.querySelector(".consent-more");
|
||||
more.href = new URL("privacy", document.baseURI).pathname;
|
||||
banner.querySelector(".consent-yes").addEventListener("click", () => setConsent("granted"));
|
||||
banner.querySelector(".consent-no").addEventListener("click", () => setConsent("denied"));
|
||||
document.body.appendChild(banner);
|
||||
}
|
||||
|
||||
// --- the withdrawal control ---------------------------------------------------
|
||||
// Any [data-consent-control] element becomes a live "Analytics: on/off — change"
|
||||
// toggle. The footer carries one on every page, so withdrawing is never more
|
||||
// than one click from wherever the visitor is.
|
||||
|
||||
function paintControls() {
|
||||
const s = consentState();
|
||||
for (const el of document.querySelectorAll("[data-consent-control]")) {
|
||||
if (signalledOptOut) {
|
||||
el.textContent = "Analytics: off (your browser asked us not to)";
|
||||
continue;
|
||||
}
|
||||
const on = s === "granted";
|
||||
el.innerHTML =
|
||||
`Analytics: ${on ? "on" : "off"} <button type="button" class="consent-toggle">`
|
||||
+ `${on ? "Turn off" : "Turn on"}</button>`;
|
||||
el.querySelector(".consent-toggle").addEventListener(
|
||||
"click", () => setConsent(on ? "denied" : "granted"));
|
||||
}
|
||||
}
|
||||
|
||||
export function initConsent() {
|
||||
// Instrumentation disabled for this environment: no banner, no control, no
|
||||
// storage. The flag is the outermost gate on everything.
|
||||
if (document.documentElement.dataset.tgEvents !== "1") return;
|
||||
paintControls();
|
||||
if (signalledOptOut) return; // already refused, at the browser level
|
||||
if (consentState() === null) showBanner();
|
||||
}
|
||||
|
||||
initConsent();
|
||||
|
|
@ -83,6 +83,10 @@
|
|||
</main>
|
||||
|
||||
<script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script>
|
||||
<!-- Product-event beacon. Inert unless the server flagged this page
|
||||
on the root element; see frontend/static/track.js. -->
|
||||
<script type="module" src="consent.js"></script>
|
||||
<script type="module" src="track.js"></script>
|
||||
<script type="module" src="day.js"></script>
|
||||
<script type="module" src="ios-install.js"></script>
|
||||
</body>
|
||||
|
|
|
|||
|
|
@ -1,40 +1,14 @@
|
|||
// Digest signup + product-event beacons. Loaded on every page (base.html.j2).
|
||||
// Digest signup form. Loaded on every page (base.html.j2).
|
||||
//
|
||||
// Both halves degrade: the form is a real <form method="post"> that works with
|
||||
// this file blocked, and the beacons are fire-and-forget with no UI depending
|
||||
// on them.
|
||||
// It degrades: the form is a real <form method="post"> that works with this
|
||||
// file blocked.
|
||||
//
|
||||
// The product-event beacon used to live here; it now lives in track.js, which
|
||||
// also owns the [data-event] click delegation this file used to do. `track` is
|
||||
// re-exported for the pages that import it from here.
|
||||
|
||||
// Backend's own origin+base (+ pinned API version), shared with account.js
|
||||
// rather than a second independent import.meta.url-derived copy (repo-split
|
||||
// Stage 5 — this file, like account.js, is always served by backend, possibly
|
||||
// from a page on a different origin once frontend/backend genuinely split).
|
||||
import { uv } from "./account.js";
|
||||
|
||||
/** Report one product event. Aggregate-only: the server takes the referrer from
|
||||
* the request itself, so nothing identifying is sent from here. */
|
||||
export function track(event) {
|
||||
try {
|
||||
const url = uv("event");
|
||||
const body = JSON.stringify({ event });
|
||||
// sendBeacon survives the page unloading, which a fetch() started on a link
|
||||
// click usually does not — that's the whole point for nav/records clicks.
|
||||
// Spec limitation: sendBeacon has no credentials/header control at all, so
|
||||
// it can never carry the auth cookie cross-origin — fine here, event
|
||||
// tracking is anonymous and needs no auth either way.
|
||||
if (navigator.sendBeacon) {
|
||||
navigator.sendBeacon(url, new Blob([body], { type: "application/json" }));
|
||||
} else {
|
||||
fetch(url, { method: "POST", body, headers: { "Content-Type": "application/json" },
|
||||
keepalive: true }).catch(() => {});
|
||||
}
|
||||
} catch { /* instrumentation must never break the page */ }
|
||||
}
|
||||
|
||||
// Anything carrying data-event reports itself when activated.
|
||||
document.addEventListener("click", (e) => {
|
||||
const el = e.target.closest?.("[data-event]");
|
||||
if (el) track(el.dataset.event);
|
||||
});
|
||||
export { track, trackLegacy } from "./track.js";
|
||||
import { trackLegacy } from "./track.js";
|
||||
|
||||
// --- digest form -------------------------------------------------------------
|
||||
for (const form of document.querySelectorAll("form[data-digest]")) {
|
||||
|
|
@ -63,7 +37,7 @@ for (const form of document.querySelectorAll("form[data-digest]")) {
|
|||
// left to read; the form is done either way.
|
||||
form.querySelector(".digest-row")?.setAttribute("hidden", "");
|
||||
form.querySelector(".digest-note")?.setAttribute("hidden", "");
|
||||
track("home.digest_signup");
|
||||
trackLegacy("home.digest_signup");
|
||||
}
|
||||
} catch {
|
||||
if (status) {
|
||||
|
|
|
|||
|
|
@ -115,6 +115,10 @@
|
|||
document.getElementById("temp-scale").innerHTML = SCALE_TEMP.map(row).join("");
|
||||
document.getElementById("rain-scale").innerHTML = SCALE_RAIN.map(row).join("");
|
||||
</script>
|
||||
<!-- Product-event beacon. Inert unless the server flagged this page
|
||||
on the root element; see frontend/static/track.js. -->
|
||||
<script type="module" src="consent.js"></script>
|
||||
<script type="module" src="track.js"></script>
|
||||
<script type="module" src="ios-install.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
|
|
|
|||
|
|
@ -48,7 +48,7 @@ function build() {
|
|||
overlay.querySelector(".mp-close").onclick = close;
|
||||
// Tapping the dimmed backdrop (outside the modal) closes the picker.
|
||||
overlay.addEventListener("pointerdown", (e) => { if (e.target === overlay) close(); });
|
||||
confirmBtn.onclick = () => finish(pin);
|
||||
confirmBtn.onclick = () => finish(pin, "map");
|
||||
|
||||
// Live suggestions (top 5, single-letter-typo tolerant) as the user types:
|
||||
// debounced so a fast typist doesn't fire a request per keystroke, aborted
|
||||
|
|
@ -131,7 +131,7 @@ function renderSug(list) {
|
|||
sugEl.hidden = true;
|
||||
searchInput.value = r.name;
|
||||
setPin(r.lat, r.lon, 10);
|
||||
finish({ lat: r.lat, lon: r.lon });
|
||||
finish({ lat: r.lat, lon: r.lon }, "search");
|
||||
};
|
||||
// Keep one highlight: hovering moves it off whatever the arrows chose.
|
||||
li.addEventListener("pointerenter", () =>
|
||||
|
|
@ -152,11 +152,14 @@ function setPin(lat, lon, zoom) {
|
|||
hintEl.textContent = "Pin set. Confirm below, or tap elsewhere to move it.";
|
||||
}
|
||||
|
||||
function finish(p) {
|
||||
// `method` tells the caller HOW the spot was chosen ("search" = picked from
|
||||
// the suggestion list, "map" = pin dropped/confirmed on the map), so a page can
|
||||
// report that dimension without the picker knowing anything about analytics.
|
||||
function finish(p, method) {
|
||||
if (!p) return;
|
||||
const cb = onPickCb;
|
||||
close();
|
||||
if (cb) cb(p.lat, p.lon);
|
||||
if (cb) cb(p.lat, p.lon, method);
|
||||
}
|
||||
|
||||
export function open(cur, onPick) {
|
||||
|
|
|
|||
|
|
@ -70,6 +70,10 @@
|
|||
</main>
|
||||
|
||||
<script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script>
|
||||
<!-- Product-event beacon. Inert unless the server flagged this page
|
||||
on the root element; see frontend/static/track.js. -->
|
||||
<script type="module" src="consent.js"></script>
|
||||
<script type="module" src="track.js"></script>
|
||||
<script type="module" src="score.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
|
|
|
|||
|
|
@ -2096,3 +2096,61 @@ details.hub-country > summary:hover { color: var(--accent); }
|
|||
font-size: 22px; line-height: 1; cursor: pointer;
|
||||
}
|
||||
.ios-a2hs-x:hover { color: var(--text); background: var(--surface-2); }
|
||||
|
||||
/* --- analytics consent ---------------------------------------------------- */
|
||||
/* A bar, never a modal: it must not block reading the page, which is the
|
||||
product. Both buttons carry identical weight — the "no" is not a muted link,
|
||||
because a refusal that is harder to click than a grant is not a free choice.
|
||||
Only rendered when instrumentation is enabled and no choice is recorded. */
|
||||
.consent-bar {
|
||||
position: fixed;
|
||||
inset: auto 0 0 0;
|
||||
z-index: 60;
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
gap: 10px 20px;
|
||||
padding: 12px 16px calc(12px + env(safe-area-inset-bottom));
|
||||
background: var(--surface-2);
|
||||
border-top: 1px solid var(--border);
|
||||
box-shadow: 0 -4px 16px rgb(0 0 0 / 0.25);
|
||||
}
|
||||
.consent-bar p { margin: 0; max-width: 60ch; font-size: 14px; color: var(--text); }
|
||||
.consent-bar a { color: var(--accent); }
|
||||
.consent-actions { display: flex; gap: 10px; }
|
||||
/* Equal size, equal prominence, both >= the 44px touch target. */
|
||||
.consent-bar button {
|
||||
min-height: 44px;
|
||||
min-width: 116px;
|
||||
padding: 10px 18px;
|
||||
font-size: 16px;
|
||||
font-weight: 600;
|
||||
border-radius: 8px;
|
||||
border: 1px solid var(--border);
|
||||
background: var(--surface);
|
||||
color: var(--text);
|
||||
cursor: pointer;
|
||||
}
|
||||
.consent-bar .consent-yes { border-color: var(--accent); color: var(--accent); }
|
||||
.consent-bar button:hover { background: var(--bg); }
|
||||
|
||||
/* The always-available withdrawal control in the footer. */
|
||||
.consent-control { display: block; margin-top: 8px; }
|
||||
.consent-control .consent-toggle {
|
||||
min-height: 32px;
|
||||
padding: 4px 10px;
|
||||
margin-left: 6px;
|
||||
font-size: 14px;
|
||||
border-radius: 6px;
|
||||
border: 1px solid var(--border);
|
||||
background: transparent;
|
||||
color: var(--accent);
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
@media (max-width: 560px) {
|
||||
.consent-bar { flex-direction: column; align-items: stretch; text-align: left; }
|
||||
.consent-actions { justify-content: stretch; }
|
||||
.consent-bar button { flex: 1; }
|
||||
}
|
||||
|
|
|
|||
|
|
@ -63,6 +63,10 @@
|
|||
</main>
|
||||
|
||||
<script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script>
|
||||
<!-- Product-event beacon. Inert unless the server flagged this page
|
||||
on the root element; see frontend/static/track.js. -->
|
||||
<script type="module" src="consent.js"></script>
|
||||
<script type="module" src="track.js"></script>
|
||||
<script type="module" src="subscriptions.js"></script>
|
||||
<script type="module" src="ios-install.js"></script>
|
||||
</body>
|
||||
|
|
|
|||
213
frontend/static/track.js
Normal file
213
frontend/static/track.js
Normal file
|
|
@ -0,0 +1,213 @@
|
|||
// Product-event beacon: what people actually do in the UI.
|
||||
//
|
||||
// Off by default. The server sets `data-tg-events="1"` on <html> only when
|
||||
// THERMOGRAPH_EVENTS is on for that environment, so with the flag unset this
|
||||
// file costs one dataset read per page and sends nothing at all.
|
||||
//
|
||||
// TWO TIERS, and the difference is the whole privacy story:
|
||||
//
|
||||
// Tier A — anonymous. Every visitor, no consent needed, because nothing is
|
||||
// stored on the device and nothing identifies anyone. The server keeps hourly
|
||||
// counts of allowlisted enums.
|
||||
//
|
||||
// Tier B — session-scoped. Adds an ephemeral id so a funnel ("did the visitor
|
||||
// who opened the picker reach a grade?") can be computed. It requires opt-in
|
||||
// consent (see consent.js) because it writes an identifier to sessionStorage.
|
||||
// Without consent, `sid()` returns "" and NOTHING is written to the device —
|
||||
// Tier A still reports, so refusing costs the product its funnels, not its
|
||||
// numbers.
|
||||
//
|
||||
// What this NEVER sends, in either tier:
|
||||
// * no stable or cross-site identifier — the Tier B id is per-tab, rotates on
|
||||
// idle/absolute caps, and is never linked to another session, device, or to
|
||||
// the signed-in account (sendBeacon cannot attach credentials cross-origin,
|
||||
// and the server does not read the session cookie on that route);
|
||||
// * no free text — never a search query, never a place name. Failed searches
|
||||
// are captured server-side only, never from here;
|
||||
// * no coordinates — `place.pick` records HOW a location was chosen, not where;
|
||||
// * no URLs — the server derives a bare referrer domain from its own headers.
|
||||
// Every field is an enum from a fixed allowlist mirrored server-side in
|
||||
// backend/core/events.py; anything else is dropped there.
|
||||
|
||||
// Backend's own origin + base + pinned API version, shared with account.js
|
||||
// rather than a second independent copy.
|
||||
import { uv } from "./account.js";
|
||||
import { hasConsent, onConsentChange, signalledOptOut } from "./consent.js";
|
||||
|
||||
const URL_ = uv("event");
|
||||
|
||||
const flagged = () => document.documentElement.dataset.tgEvents === "1";
|
||||
|
||||
// Tier A: the flag, and an explicit browser-level opt-out signal. GPC/DNT are
|
||||
// honoured here too even though Tier A needs no consent — a visitor who has
|
||||
// asked not to be measured should not be measured, and the resulting downward
|
||||
// bias is small and worth it.
|
||||
const enabled = () => flagged() && !signalledOptOut;
|
||||
|
||||
// --- Tier B: the ephemeral session id ------------------------------------------
|
||||
// sessionStorage, not localStorage and not a cookie:
|
||||
// * localStorage would persist across sessions and make the id a stable
|
||||
// pseudonym — the exact thing being avoided;
|
||||
// * a cookie would ride along on every request including static assets, be
|
||||
// readable server-side on routes that have nothing to do with analytics,
|
||||
// and buy nothing here.
|
||||
// sessionStorage is per-tab and dies with the tab. Consequence, stated honestly:
|
||||
// a visitor who opens Compare in a second tab is two sessions, so funnels
|
||||
// UNDERCOUNT completion. That is the error direction to prefer.
|
||||
const SID_KEY = "thermograph:sid";
|
||||
const IDLE_MS = 30 * 60 * 1000; // rotate after 30 minutes of inactivity
|
||||
const MAX_MS = 2 * 60 * 60 * 1000; // and after 2 hours regardless — a hard cap
|
||||
// on how long any one behavioural sequence
|
||||
// can possibly be
|
||||
const MAX_SEQ = 200; // mirrors events.MAX_SEQ server-side
|
||||
|
||||
function mintId() {
|
||||
// 16 random bytes -> 22 base64url chars. Matches the server's _SESSION_RE
|
||||
// exactly; anything else in this field is dropped whole rather than trimmed.
|
||||
const b = new Uint8Array(16);
|
||||
crypto.getRandomValues(b);
|
||||
return btoa(String.fromCharCode(...b)).replace(/\+/g, "-").replace(/\//g, "_").replace(/=+$/, "");
|
||||
}
|
||||
|
||||
/** The current session id, or "" when there must not be one. Minting is the
|
||||
* moment something is first written to the device, so it happens ONLY here and
|
||||
* ONLY behind hasConsent(). */
|
||||
function sid() {
|
||||
if (!flagged() || !hasConsent()) return "";
|
||||
try {
|
||||
const now = Date.now();
|
||||
let rec = null;
|
||||
try { rec = JSON.parse(sessionStorage.getItem(SID_KEY) || "null"); } catch { rec = null; }
|
||||
if (!rec || !rec.id || now - rec.last > IDLE_MS || now - rec.born > MAX_MS) {
|
||||
rec = { id: mintId(), born: now, last: now, n: 0 };
|
||||
}
|
||||
rec.last = now;
|
||||
rec.n = Math.min((rec.n || 0) + 1, MAX_SEQ + 1);
|
||||
sessionStorage.setItem(SID_KEY, JSON.stringify(rec));
|
||||
return rec.n > MAX_SEQ ? "" : rec.id; // past the cap: back to Tier A only
|
||||
} catch { return ""; } // private mode / storage disabled: Tier A only
|
||||
}
|
||||
|
||||
function seqOf() {
|
||||
try {
|
||||
const rec = JSON.parse(sessionStorage.getItem(SID_KEY) || "null");
|
||||
return rec && typeof rec.n === "number" ? rec.n : 0;
|
||||
} catch { return 0; }
|
||||
}
|
||||
|
||||
export function dropSession() {
|
||||
try { sessionStorage.removeItem(SID_KEY); } catch { /* nothing to drop */ }
|
||||
}
|
||||
|
||||
// Withdrawing consent must take effect immediately, not at the next page load:
|
||||
// drop the live id and discard anything queued but not yet sent.
|
||||
onConsentChange((state) => {
|
||||
if (state !== "granted") { queue = []; dropSession(); }
|
||||
});
|
||||
|
||||
// Batch rather than one request per interaction: chart-metric taps and slider
|
||||
// drags arrive in bursts, and ~0.03 req/s of real traffic does not need a
|
||||
// request each. Flushed on a short timer and unconditionally when the page is
|
||||
// hidden or unloaded, which is the case a plain fetch() would lose.
|
||||
const MAX_BATCH = 20; // mirrors events.MAX_BATCH server-side
|
||||
const FLUSH_MS = 4000;
|
||||
let queue = [];
|
||||
let timer = 0;
|
||||
|
||||
function send(items, session) {
|
||||
try {
|
||||
const body = JSON.stringify(session ? { events: items, session } : { events: items });
|
||||
if (navigator.sendBeacon) {
|
||||
// sendBeacon survives the page unloading, which a fetch() started on a
|
||||
// link click usually does not — that's the whole point for nav clicks.
|
||||
navigator.sendBeacon(URL_, new Blob([body], { type: "application/json" }));
|
||||
} else {
|
||||
fetch(URL_, {
|
||||
method: "POST", body, keepalive: true,
|
||||
headers: { "Content-Type": "application/json" },
|
||||
}).catch(() => {});
|
||||
}
|
||||
} catch { /* instrumentation must never break the page */ }
|
||||
}
|
||||
|
||||
export function flush() {
|
||||
clearTimeout(timer);
|
||||
timer = 0;
|
||||
if (!queue.length) return;
|
||||
const items = queue;
|
||||
queue = [];
|
||||
// Resolved at flush time, not at queue time: if consent was withdrawn between
|
||||
// the two, the id is already gone and the batch goes out anonymously. Strip
|
||||
// the carrier field BEFORE serialising — it must never reach the wire.
|
||||
const session = hasConsent() ? (items[0].__sid || "") : "";
|
||||
for (const it of items) delete it.__sid;
|
||||
send(items, session);
|
||||
}
|
||||
|
||||
/** Report one product event. `props` may carry only {view, prop, value}, all
|
||||
* enums — see backend/core/events.py SCHEMA for the allowlists. */
|
||||
export function track(event, props) {
|
||||
try {
|
||||
if (!enabled() || !event) return;
|
||||
// sid() both reads and advances the counter, so call it exactly once per
|
||||
// event. It returns "" (and writes nothing at all) without consent.
|
||||
const id = sid();
|
||||
const item = props ? { event, props } : { event };
|
||||
if (id) { item.seq = seqOf(); item.__sid = id; }
|
||||
queue.push(item);
|
||||
if (queue.length >= MAX_BATCH) { flush(); return; }
|
||||
if (!timer) timer = setTimeout(flush, FLUSH_MS);
|
||||
} catch { /* never break the page */ }
|
||||
}
|
||||
|
||||
// The four aggregate counters that already ship in production (metrics.EVENTS +
|
||||
// home.nav_*). They predate the flag and predate the schema, and the ops
|
||||
// endpoint's numbers must not silently reset when this file lands — so they keep
|
||||
// their old unconditional, unbatched behaviour until they are migrated onto the
|
||||
// schema above and the old names are retired. Everything NEW goes through
|
||||
// track() and is flag-gated.
|
||||
const LEGACY = /^home\.(locate|digest_signup|records_click|share|nav_[a-z]+)$/;
|
||||
|
||||
export function trackLegacy(event) {
|
||||
try {
|
||||
if (signalledOptOut || !event) return;
|
||||
send([{ event }]); // anonymous, unbatched, never session-scoped
|
||||
} catch { /* never break the page */ }
|
||||
}
|
||||
|
||||
/** Which view this page is, for the `view` dimension. Derived from the path so
|
||||
* no page has to declare it twice; the server re-checks it against VIEWS. */
|
||||
export function currentView() {
|
||||
const p = location.pathname.replace(/\/+$/, "");
|
||||
const last = p.slice(p.lastIndexOf("/") + 1);
|
||||
if (!last) return "home";
|
||||
if (["calendar", "day", "compare", "score", "legend", "privacy", "about"].includes(last)) return last;
|
||||
if (last === "alerts") return "alerts";
|
||||
if (p.includes("/climate/")) return p.split("/climate/")[1].includes("/") ? "month" : "city";
|
||||
if (p.endsWith("/climate")) return "cities";
|
||||
if (p.includes("/glossary")) return "glossary";
|
||||
if (last === "records") return "records";
|
||||
return "home";
|
||||
}
|
||||
|
||||
// pagehide covers the bfcache/navigation case; visibilitychange covers a tab
|
||||
// switch or the phone locking. Both fire where beforeunload is unreliable.
|
||||
addEventListener("pagehide", flush);
|
||||
addEventListener("visibilitychange", () => { if (document.hidden) flush(); });
|
||||
|
||||
// Anything carrying data-event reports itself when activated. Optional
|
||||
// data-event-prop / data-event-value carry the two extra dimensions, so a
|
||||
// server-rendered link needs no JS of its own.
|
||||
document.addEventListener("click", (e) => {
|
||||
const el = e.target.closest?.("[data-event]");
|
||||
if (!el) return;
|
||||
const d = el.dataset;
|
||||
if (LEGACY.test(d.event)) { trackLegacy(d.event); return; }
|
||||
track(d.event, {
|
||||
view: d.eventView || currentView(),
|
||||
prop: d.eventProp || "",
|
||||
value: d.eventValue || "",
|
||||
});
|
||||
// A click that navigates away must not lose its own event.
|
||||
if (el.tagName === "A") flush();
|
||||
});
|
||||
|
|
@ -159,8 +159,13 @@ function syncUnitToggle(wrap) {
|
|||
// Labelled for what it actually switches now — temperature, precipitation and
|
||||
// wind — even though the buttons read °F/°C.
|
||||
wrap.setAttribute("aria-label", "Units");
|
||||
wrap.innerHTML = '<button type="button" data-unit="F">°F</button>'
|
||||
+ '<button type="button" data-unit="C">°C</button>';
|
||||
// data-event/-prop/-value are picked up by track.js's global click delegation,
|
||||
// so this file needs no import and no analytics code of its own. Which unit
|
||||
// people actually switch TO is the number that would justify changing the
|
||||
// locale-derived default.
|
||||
wrap.innerHTML =
|
||||
'<button type="button" data-unit="F" data-event="view.control" data-event-prop="unit" data-event-value="F">°F</button>'
|
||||
+ '<button type="button" data-unit="C" data-event="view.control" data-event-prop="unit" data-event-value="C">°C</button>';
|
||||
wrap.addEventListener("click", (e) => {
|
||||
const b = e.target.closest("button[data-unit]");
|
||||
if (b) setUnit(b.dataset.unit);
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<html lang="en"{% if events_enabled %} data-tg-events="1"{% endif %}>
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
|
|
@ -94,7 +94,12 @@
|
|||
<a href="{{ base }}/privacy">Privacy</a>
|
||||
<a href="{{ base }}/">Open the tool</a>
|
||||
</nav>
|
||||
<p class="muted">Free. No ads. No tracking. No account needed. Built by one person in the open.</p>
|
||||
<p class="muted">Free. No ads. No third-party trackers. No account needed. Built by one
|
||||
person in the open.</p>
|
||||
{# Withdrawing consent must be as easy as giving it and reachable from
|
||||
anywhere, so the control lives in the site-wide footer. consent.js
|
||||
fills it in and leaves it empty when instrumentation is disabled. #}
|
||||
<p class="muted consent-control" data-consent-control></p>
|
||||
<p class="muted">Thermograph grades weather by its percentile against ~45 years of local
|
||||
climate history. Data: ERA5 via Open-Meteo · NASA POWER · MET Norway.</p>
|
||||
<p class="muted">Made by <a href="https://emigriffith.dev">emigriffith.dev</a>
|
||||
|
|
@ -107,6 +112,7 @@
|
|||
<script type="module" src="{{ asset_base }}/account.js"></script>
|
||||
<script type="module" src="{{ asset_base }}/climate.js"></script>
|
||||
{% endblock %}
|
||||
<script type="module" src="{{ asset_base }}/consent.js"></script>
|
||||
<script type="module" src="{{ asset_base }}/digest.js"></script>
|
||||
<script type="module" src="{{ asset_base }}/ios-install.js"></script>
|
||||
</body>
|
||||
|
|
|
|||
|
|
@ -99,7 +99,7 @@
|
|||
{# Sits outside #panel on purpose: app.js rewrites results.innerHTML
|
||||
wholesale on every grade, so anything inside it would be destroyed. #}
|
||||
<p class="stance-line muted">
|
||||
Free. No ads. No tracking. No account needed.
|
||||
Free. No ads. No third-party trackers. No account needed.
|
||||
<a href="{{ base }}/about">How we work →</a>
|
||||
</p>
|
||||
|
||||
|
|
|
|||
|
|
@ -16,15 +16,63 @@
|
|||
learns where you are is if you press <b>Use my location</b>, which asks your browser for
|
||||
permission. You can decline, and picking a place on the map works just as well.</p>
|
||||
<p>Whichever way you choose a place, the coordinates are sent to the server only as part of
|
||||
the ordinary request for that grid square's climate data, and are not logged beyond the
|
||||
normal web-server request handling every site does. The place you picked is remembered in
|
||||
your own browser's local storage, not on the server, so clearing your browser data
|
||||
forgets it.</p>
|
||||
the ordinary request for that grid square's climate data. The place you picked is
|
||||
remembered in your own browser's local storage, not on the server, so clearing your
|
||||
browser data forgets it. <b>The coordinates are never recorded in the usage statistics
|
||||
below</b> — those record only <i>how</i> you picked a place (map, search, or your
|
||||
browser's location), never <i>where</i>.</p>
|
||||
|
||||
<h2>Analytics</h2>
|
||||
<p>There is no analytics library, no cookies for tracking, and no per-visitor identifier.
|
||||
The server keeps simple aggregate counts (how many people opened a page or tapped a
|
||||
button, and which site referred them) with nothing tying any of it to an individual.</p>
|
||||
<h2>Usage statistics</h2>
|
||||
<p>There is no third-party analytics service, no advertising tag, and nothing that follows
|
||||
you to other sites. What the site does keep is in two parts, and only the second one
|
||||
asks your permission.</p>
|
||||
|
||||
<h3>Anonymous counts — always on</h3>
|
||||
<p>The server keeps simple counts of things like "the calendar was opened 40 times this
|
||||
hour" or "a search found nothing 12 times today", along with which site referred the
|
||||
visit (the domain only, never the full link). No identifier of any kind is attached,
|
||||
nothing is stored on your device, and there is no way to tell one visitor's counts from
|
||||
another's — so there is nothing here that could be traced back to you. These counts are
|
||||
kept for up to 400 days.</p>
|
||||
|
||||
<h3>Session statistics — only if you say yes</h3>
|
||||
<p>To answer questions like "do people who open the location picker actually get a
|
||||
forecast, or give up half way?", the site has to see several actions as belonging to the
|
||||
same visit. That means putting a random identifier in your browser, so
|
||||
<b>we ask first, and it stays off unless you agree</b>.</p>
|
||||
<p>If you agree:</p>
|
||||
<ul>
|
||||
<li>a random number is generated and kept in your browser's <b>session storage</b>, for
|
||||
the current tab only. It is not a cookie, and it is not sent to any other site;</li>
|
||||
<li>it changes after 30 minutes of inactivity, and after 2 hours regardless, so it cannot
|
||||
follow you across a day, a device, or a return visit;</li>
|
||||
<li>it disappears when you close the tab;</li>
|
||||
<li>it records which views and controls you used, from a short fixed list — never what you
|
||||
typed, never where you are, never which articles you read;</li>
|
||||
<li>it is <b>never linked to your account</b>, even when you are signed in, and never
|
||||
linked to your email address or your IP address;</li>
|
||||
<li>the records are deleted after <b>30 days</b>.</li>
|
||||
</ul>
|
||||
<p>You can change your mind at any time with the <b>Analytics</b> control at the bottom of
|
||||
every page. Turning it off deletes the identifier from your browser immediately and stops
|
||||
any further recording. Because the records are deliberately not linked to you, we cannot
|
||||
go back and find your past ones to delete — which is part of why they are kept for such a
|
||||
short time.</p>
|
||||
<p>If your browser sends a <b>Global Privacy Control</b> or <b>Do Not Track</b> signal, the
|
||||
site treats that as a no, does not ask, and records nothing extra.</p>
|
||||
|
||||
<h3>Searches that found nothing</h3>
|
||||
<p>When you search for a place and the site finds no match, the text you typed may be kept
|
||||
so that missing places can be added. It is stored on its own, with nothing else about you
|
||||
attached, and only if it looks like a place name — anything containing an email address,
|
||||
a web address or a long run of digits is discarded rather than stored. Anything only one
|
||||
or two people ever searched for is deleted within a week; the rest is deleted after
|
||||
90 days. Searches that <i>did</i> find a place are never stored as text.</p>
|
||||
|
||||
<h2>Server logs</h2>
|
||||
<p>As on every website, the server records that a request happened — the page, the time and
|
||||
the result. Network addresses are shortened before they are written down, so an individual
|
||||
device cannot be picked out of the logs, and the logs are deleted after 30 days.</p>
|
||||
|
||||
<h2>Email</h2>
|
||||
<p>If you sign up for the monthly digest, your address is used for that digest and nothing
|
||||
|
|
@ -32,8 +80,10 @@
|
|||
|
||||
<h2>Accounts</h2>
|
||||
<p>An account is optional and only exists to hold weather alerts you have asked for. It
|
||||
stores your email address and the places you are watching.</p>
|
||||
stores your email address and the places you are watching. None of the usage statistics
|
||||
above are connected to it.</p>
|
||||
|
||||
<p class="muted">Questions: see <a href="{{ base }}/about">About & methodology</a>.</p>
|
||||
<p class="muted consent-control" data-consent-control></p>
|
||||
</article>
|
||||
{% endblock %}
|
||||
|
|
|
|||
|
|
@ -32,3 +32,84 @@ def test_old_static_index_is_gone(client):
|
|||
backend/tests/web/test_homepage.py, repo-split Stage 7a -- frontend's own
|
||||
StaticFiles mount is what actually answers this now)."""
|
||||
assert client.get(f"{B}/index.html").status_code == 404
|
||||
|
||||
|
||||
# --- product-event instrumentation flag ---------------------------------------
|
||||
# track.js sends nothing unless it finds data-tg-events on <html>. The whole
|
||||
# rollout story rests on that stamp being absent by default, so assert both
|
||||
# halves: off unless THERMOGRAPH_EVENTS is set, and applied to BOTH HTML paths
|
||||
# (the Jinja-rendered content pages and the static SPA shells).
|
||||
|
||||
def test_event_flag_is_off_by_default_on_every_html_path(client):
|
||||
for path in ("/", "/calendar", "/day", "/score", "/compare", "/alerts"):
|
||||
r = client.get(f"{B}{path}")
|
||||
assert r.status_code == 200, path
|
||||
assert "data-tg-events" not in r.text, path
|
||||
|
||||
|
||||
def test_event_flag_stamps_both_html_paths_when_enabled(monkeypatch):
|
||||
"""Rebuilt in-process rather than reusing `client`: the flag is read at
|
||||
import and the SPA shells memoize their prepped template."""
|
||||
import importlib
|
||||
import content as content_mod
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
monkeypatch.setenv("THERMOGRAPH_EVENTS", "1")
|
||||
content_mod = importlib.reload(content_mod)
|
||||
assert content_mod.EVENTS_ENABLED
|
||||
import app as app_mod
|
||||
app_mod = importlib.reload(app_mod)
|
||||
flagged = TestClient(app_mod.app)
|
||||
for path in ("/calendar", "/compare"):
|
||||
assert 'data-tg-events="1"' in flagged.get(f"{B}{path}").text, path
|
||||
# Leave the modules as the rest of the suite expects to find them.
|
||||
monkeypatch.delenv("THERMOGRAPH_EVENTS")
|
||||
importlib.reload(content_mod)
|
||||
importlib.reload(app_mod)
|
||||
|
||||
|
||||
def test_consent_control_is_present_on_every_page(client):
|
||||
"""Withdrawal must be reachable from anywhere, so the footer carries the
|
||||
control on every server-rendered page. consent.js leaves it empty when
|
||||
instrumentation is disabled, so it costs nothing when off."""
|
||||
for path in ("/", "/privacy", "/climate"):
|
||||
r = client.get(f"{B}{path}")
|
||||
assert r.status_code == 200, path
|
||||
assert "data-consent-control" in r.text, path
|
||||
|
||||
|
||||
def test_consent_script_loads_before_the_tracker(client):
|
||||
"""consent.js owns the banner and the opt-in gate; track.js imports it. It is
|
||||
listed first so the gate is installed even on a page whose tracker chain
|
||||
never loads."""
|
||||
for path, after in (("/", "digest.js"), ("/calendar", "track.js")):
|
||||
body = client.get(f"{B}{path}").text
|
||||
tag = lambda name: body.index(f'src="{B}/{name}"' if path == "/" else f'src="{name}"')
|
||||
assert tag("consent.js") < tag(after), path
|
||||
|
||||
|
||||
def test_privacy_page_describes_what_is_actually_collected(client):
|
||||
"""The page is a public commitment; if this test has to change, the copy and
|
||||
the code have drifted apart. Asserts the load-bearing promises only."""
|
||||
body = client.get(f"{B}/privacy").text
|
||||
for promise in (
|
||||
"session storage", # where the id lives
|
||||
"30 minutes", # idle rotation
|
||||
"2 hours", # absolute cap
|
||||
"never linked to your account",
|
||||
"30 days", # session-row retention
|
||||
"400 days", # anonymous aggregate retention
|
||||
"90 days", # failed-search retention
|
||||
"Global Privacy Control",
|
||||
"only if you say yes", # opt-in, not opt-out
|
||||
):
|
||||
assert promise in body, promise
|
||||
# The v1 claim that this change makes untrue must be gone.
|
||||
assert "no per-visitor identifier" not in body
|
||||
|
||||
|
||||
def test_footer_no_longer_claims_no_tracking(client):
|
||||
"""A consented session id is an identifier; the blanket claim would be a lie."""
|
||||
body = client.get(f"{B}/").text
|
||||
assert "No tracking." not in body
|
||||
assert "No third-party trackers." in body
|
||||
|
|
|
|||
Loading…
Reference in a new issue