35b58300cb
Three operator decisions change v1's premises, so the design is now two tiers with separate flags: Tier A (THERMOGRAPH_EVENTS) is v1 unchanged -- anonymous hourly aggregates, no identifier, nothing stored on the device, no consent needed, runs for everyone. It stays the primary signal precisely so a poor consent rate cannot take the numbers down with it, and so Tier B's rates can be calibrated against it to measure the consent bias rather than ignoring it. Tier B (THERMOGRAPH_EVENT_SESSIONS) adds an ephemeral per-tab id -- 16 random bytes in sessionStorage, rotating on a 30-minute idle and a 2-hour absolute cap, capped at 200 events, never linked to another session, device, or to the account (api_event does not read the auth cookie and there is no user column). Rows land in a 30-day hypertable with minute-granularity timestamps and an in-session sequence number instead of precise clock times. Storing an identifier engages ePrivacy Art. 5(3) and analytics is not strictly necessary, so it is gated on opt-in consent: an equal-weight banner that mints nothing before "Allow", one-click withdrawal in every footer that drops the live id immediately, and GPC/DNT treated as a refusal already given. THERMOGRAPH_SEARCH_MISS captures zero-result search text server-side in api_suggest only -- normalised, rejected outright on any personal-data smell, stored as a per-day count, and pruned below a three-person floor after a week. The privacy page is rewritten rather than deferred: "no per-visitor identifier" becomes false the moment Tier B ships. Tests assert the load-bearing promises so the copy and the code cannot drift apart silently. Raw-IP truncation is NOT implemented here (the logging pipeline owns it) but UI-EVENTS.md states what the app side must do, and records that Caddy's default JSON log already stores full request URIs -- so every search query is in Loki with the client IP today, which the search-miss mitigations depend on fixing. UI-EVENTS.md carries the v1-to-v2 diff, the consent reasoning including why legitimate interest is not available, and an explicit argument that the session identifier is the wrong trade at this traffic volume.
98 lines
4.1 KiB
Python
98 lines
4.1 KiB
Python
"""UI events tier B: consented session rows + failed-search text
|
|
|
|
Two tables, both created together because they arrived from the same pair of
|
|
operator decisions (see UI-EVENTS.md §6):
|
|
|
|
* ``ui_event_session`` — one row per event, carrying an ephemeral per-tab session
|
|
id, so a funnel can be computed. This is a **behavioural sequence dataset**:
|
|
the thing ``ui_event_hourly`` was deliberately shaped to avoid. It therefore
|
|
gets the shortest retention in the system (30 days), minute-granularity
|
|
timestamps rather than precise ones, and an in-session ``seq`` to carry order.
|
|
Only ever written for a visitor who gave opt-in consent.
|
|
* ``ui_search_miss`` — normalised zero-result search text, the only free-text
|
|
field anywhere in the instrumentation, captured server-side only. Stored as a
|
|
per-day count rather than a row per search, and pruned below a k-anonymity
|
|
floor (see ``core/events.py`` SEARCH_MISS_MIN_COUNT) so a string one person
|
|
typed once does not persist.
|
|
|
|
Retention numbers live here, not in a hand-typed policy on the box, so they are
|
|
reviewable in the repo and move with the code.
|
|
|
|
Postgres-only: guarded to no-op on the SQLite bind (tests / CI / offline
|
|
Alembic), where both sinks are inert.
|
|
|
|
Revision ID: 0004_ui_event_sessions
|
|
Revises: 0003_ui_events
|
|
Create Date: 2026-07-23
|
|
"""
|
|
from alembic import op
|
|
|
|
revision = "0004_ui_event_sessions"
|
|
down_revision = "0003_ui_events"
|
|
branch_labels = None
|
|
depends_on = None
|
|
|
|
# The behavioural-sequence table. Short on purpose: a funnel question is answered
|
|
# in days-to-weeks, and the derived answer should be rolled up into the anonymous
|
|
# aggregate rather than kept as raw sequences.
|
|
SESSION_RETENTION_DAYS = 30
|
|
|
|
# Free-text search misses. Long enough to spot a seasonal gap in the place index,
|
|
# short enough to be defensible for a field a person can type anything into.
|
|
SEARCH_MISS_RETENTION_DAYS = 90
|
|
|
|
|
|
def upgrade() -> None:
|
|
if op.get_bind().dialect.name != "postgresql":
|
|
return # SQLite fallback (tests/CI): both sinks are no-ops there.
|
|
|
|
op.execute("CREATE EXTENSION IF NOT EXISTS timescaledb")
|
|
|
|
# No IP column, no user-agent column, no account-id column, and no unique
|
|
# constraint that would require one: there is deliberately nothing here that
|
|
# could re-identify a session, and nothing to join it to.
|
|
op.execute("""
|
|
CREATE TABLE ui_event_session (
|
|
ts TIMESTAMPTZ NOT NULL,
|
|
session TEXT NOT NULL,
|
|
seq INTEGER NOT NULL,
|
|
event TEXT NOT NULL,
|
|
view TEXT NOT NULL DEFAULT '',
|
|
prop TEXT NOT NULL DEFAULT '',
|
|
value TEXT NOT NULL DEFAULT '',
|
|
referrer TEXT NOT NULL DEFAULT ''
|
|
)
|
|
""")
|
|
# 1-day chunks: the table is dropped a chunk at a time by the retention
|
|
# policy, and every query against it is "the last N days".
|
|
op.execute(
|
|
"SELECT create_hypertable('ui_event_session', by_range('ts', INTERVAL '1 day'))")
|
|
op.execute(
|
|
f"SELECT add_retention_policy('ui_event_session', "
|
|
f"INTERVAL '{SESSION_RETENTION_DAYS} days')")
|
|
# The only access pattern is "walk one session in order" (funnels) and
|
|
# "everything in a window" (the rollup).
|
|
op.execute("CREATE INDEX ui_event_session_sid_idx ON ui_event_session (session, seq)")
|
|
|
|
# Failed searches: a per-day count, never a row per search.
|
|
op.execute("""
|
|
CREATE TABLE ui_search_miss (
|
|
day DATE NOT NULL,
|
|
q TEXT NOT NULL,
|
|
count BIGINT NOT NULL DEFAULT 0,
|
|
PRIMARY KEY (day, q)
|
|
)
|
|
""")
|
|
# Plain table, not a hypertable: it is tiny (bounded by distinct failed
|
|
# queries per day) and the prune is a predicate on `count`, not a chunk drop,
|
|
# so time-partitioning buys nothing. Retention is enforced by the same daily
|
|
# job that applies the k-anonymity prune -- see core/events.py
|
|
# PRUNE_SEARCH_MISS_SQL and UI-EVENTS.md.
|
|
op.execute("CREATE INDEX ui_search_miss_day_idx ON ui_search_miss (day)")
|
|
|
|
|
|
def downgrade() -> None:
|
|
if op.get_bind().dialect.name != "postgresql":
|
|
return
|
|
op.execute("DROP TABLE IF EXISTS ui_search_miss")
|
|
op.execute("DROP TABLE IF EXISTS ui_event_session")
|