0ddc457d8c
Extends the existing /api/v2/event beacon into a small, typed, durable event schema so the interactive views can answer product questions the request log cannot: whether a visitor ever gets a location, whether search finds anything, which controls and views earn their maintenance, and which dead ends people actually hit. Seven events with three enum dimension slots, stored as hourly aggregates in a TimescaleDB hypertable plus one JSONL line per event for the 30-day Loki view. No row per interaction and no column an identifier could go in: no cookie, no session id, no user id, no IP, no coordinates, no free text, no URLs. The IP is a per-minute rate-limit key and nothing else. Abuse resistance for a public endpoint on a 60%-crawler site: closed allowlist (unknown names collapse to one bucket), 4 KB streaming body cap, per-IP ceiling raised to 120/min (30 was sized for four coarse events and would have silently truncated a batched stream), soft Sec-Fetch-Site first-party check, uniform 204. Ships inert -- THERMOGRAPH_EVENTS gates both the recorder and the flag stamp on <html> that makes the client send anything, and is unset everywhere. See UI-EVENTS.md for the schema, the rejected transport/storage alternatives, and the privacy decisions that must be settled before the flag is turned on.
69 lines
2.8 KiB
Python
69 lines
2.8 KiB
Python
"""UI product events: hourly aggregate hypertable
|
|
|
|
One row per (hour, event, view, prop, value, referrer) — never one row per
|
|
interaction. See ``backend/core/events.py`` for the schema and the reasoning:
|
|
the table has no column an identifier could be written into, so it cannot hold
|
|
a sequence of one person's actions even in principle.
|
|
|
|
TimescaleDB hypertable on ``bucket`` with a 30-day chunk interval (the read
|
|
pattern is "last N weeks, grouped by event"), plus a retention policy so old
|
|
buckets drop themselves. ``RETENTION_DAYS`` is the operator-set retention limit
|
|
— change it here, not by hand on the box, so the deployed value is reviewable.
|
|
|
|
Postgres-only: guarded to no-op on the SQLite bind (tests / CI / offline
|
|
Alembic), where events.py's aggregate sink is inert and only the JSONL activity
|
|
line is written.
|
|
|
|
Revision ID: 0003_ui_events
|
|
Revises: 0002_climate_hypertables
|
|
Create Date: 2026-07-23
|
|
"""
|
|
from alembic import op
|
|
|
|
revision = "0003_ui_events"
|
|
down_revision = "0002_climate_hypertables"
|
|
branch_labels = None
|
|
depends_on = None
|
|
|
|
# Aggregate rows carry no personal data, so this is a storage/usefulness choice
|
|
# rather than a GDPR one. 400 days keeps a full year-over-year comparison.
|
|
RETENTION_DAYS = 400
|
|
|
|
|
|
def upgrade() -> None:
|
|
if op.get_bind().dialect.name != "postgresql":
|
|
return # SQLite fallback (tests/CI): the aggregate sink is a no-op there.
|
|
|
|
op.execute("CREATE EXTENSION IF NOT EXISTS timescaledb")
|
|
|
|
# Every dimension is NOT NULL DEFAULT '' rather than nullable: '' is a real
|
|
# value ("this event does not use this slot"), and it keeps the primary key
|
|
# usable (NULLs would defeat the ON CONFLICT upsert).
|
|
op.execute("""
|
|
CREATE TABLE ui_event_hourly (
|
|
bucket TIMESTAMPTZ NOT NULL,
|
|
event TEXT NOT NULL,
|
|
view TEXT NOT NULL DEFAULT '',
|
|
prop TEXT NOT NULL DEFAULT '',
|
|
value TEXT NOT NULL DEFAULT '',
|
|
referrer TEXT NOT NULL DEFAULT '',
|
|
count BIGINT NOT NULL DEFAULT 0,
|
|
PRIMARY KEY (bucket, event, view, prop, value, referrer)
|
|
)
|
|
""")
|
|
# The PK includes the range-partition column (bucket), which TimescaleDB
|
|
# requires of any unique index on a hypertable.
|
|
op.execute(
|
|
"SELECT create_hypertable('ui_event_hourly', "
|
|
"by_range('bucket', INTERVAL '30 days'))")
|
|
op.execute(
|
|
f"SELECT add_retention_policy('ui_event_hourly', INTERVAL '{RETENTION_DAYS} days')")
|
|
|
|
# The dashboard query is "one event over time", so index it that way.
|
|
op.execute("CREATE INDEX ui_event_hourly_event_idx ON ui_event_hourly (event, bucket DESC)")
|
|
|
|
|
|
def downgrade() -> None:
|
|
if op.get_bind().dialect.name != "postgresql":
|
|
return
|
|
op.execute("DROP TABLE IF EXISTS ui_event_hourly")
|