490 lines
28 KiB
Markdown
490 lines
28 KiB
Markdown
|
|
# Approximate-location fallback — design + prototype
|
|||
|
|
|
|||
|
|
Status: **prototype on `feat/geoip-approx-location`, feature-flagged OFF, not
|
|||
|
|
deployed and not merged.** Nothing in this branch changes production behaviour
|
|||
|
|
until `THERMOGRAPH_GEOIP` is set *and* a database file exists on the host.
|
|||
|
|
|
|||
|
|
This doc is the decision record. If it survives review it should move to
|
|||
|
|
`thermograph-docs` (per the root `CLAUDE.md`: cross-cutting decision docs live
|
|||
|
|
there); it sits here for now so the reasoning ships with the code it describes.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 1. The problem
|
|||
|
|
|
|||
|
|
Location today comes from exactly one source: `navigator.geolocation.getCurrentPosition`
|
|||
|
|
in `frontend/static/app.js`, gated on `window.isSecureContext`. If the visitor
|
|||
|
|
declines the permission prompt, the prompt times out, or the browser won't offer
|
|||
|
|
geolocation at all, the site has **no location** and the copy says "Pick a spot
|
|||
|
|
on the map instead."
|
|||
|
|
|
|||
|
|
That is a dead end for the exact visitor the homepage is written for — someone
|
|||
|
|
who arrived to answer "how unusual is *my* weather?" and has just been handed a
|
|||
|
|
world map. This change gives that person something local, honestly labelled.
|
|||
|
|
|
|||
|
|
## 2. What was built
|
|||
|
|
|
|||
|
|
| Piece | File |
|
|||
|
|
| --- | --- |
|
|||
|
|
| Lookup module (flag, parsing, rejection rules, attribution) | `backend/data/geoip.py` |
|
|||
|
|
| Endpoint `GET /api/v2/geoip` | `backend/web/app.py` (`api_geoip`) |
|
|||
|
|
| Access-log exclusion + traffic category | `backend/web/app.py`, `backend/core/metrics.py` |
|
|||
|
|
| Event names `home.geoip_shown` / `home.geoip_fixed` | `backend/core/metrics.py` |
|
|||
|
|
| Client fetch + suggestion banner | `frontend/static/geoip.js` |
|
|||
|
|
| Wiring, non-persistence, honest headings | `frontend/static/app.js` |
|
|||
|
|
| Banner container (empty, hidden) | `frontend/templates/home.html.j2` |
|
|||
|
|
| Banner styles | `frontend/static/style.css` |
|
|||
|
|
| Flag-aware privacy copy | `frontend/content.py`, `frontend/templates/privacy.html.j2` |
|
|||
|
|
| Host refresh job + timer | `infra/deploy/geoip-refresh.{sh,service,timer}` |
|
|||
|
|
| Read-only DB mount + env plumbing | `infra/docker-compose.yml`, `infra/deploy/thermograph.env.example` |
|
|||
|
|
| Tests | `backend/tests/data/test_geoip.py`, `backend/tests/web/test_geoip_api.py`, `frontend/tests/unit/test_geoip_ui.py` |
|
|||
|
|
|
|||
|
|
`cd backend && make test` → **449 passed, 8 skipped** (was 386/7).
|
|||
|
|
`cd frontend && make test-unit` → **41 passed**.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 3. Database choice — recommendation: **DB-IP IP to City Lite**
|
|||
|
|
|
|||
|
|
| | **DB-IP IP to City Lite** | MaxMind GeoLite2 City | IP2Location LITE DB5/DB11 |
|
|||
|
|
| --- | --- | --- | --- |
|
|||
|
|
| Licence | **CC BY 4.0** | GeoLite End User License Agreement (incorporates CC BY-SA 4.0 terms) | CC BY-SA 4.0 / "free with attribution" |
|
|||
|
|
| Account needed | **No** — direct download URL | **Yes** — account + licence key | **Yes** — free account |
|
|||
|
|
| Secret to manage | **None** | Licence key (→ SOPS vault, → CI, → rotation) | Account credentials |
|
|||
|
|
| Attribution | `<a href="https://db-ip.com">IP Geolocation by DB-IP</a>` on pages showing results | "This product includes GeoLite Data created by MaxMind, available from https://www.maxmind.com" | "uses the IP2Location LITE database for IP geolocation", linked |
|
|||
|
|
| Retention obligation | none | **must delete a database within 30 days of a newer release**; 30 downloads/day cap | none |
|
|||
|
|
| Update cadence | monthly | weekly (Tue/Fri) | monthly |
|
|||
|
|
| Size (current release) | 125 MB MMDB (685 MB CSV), ~8.05 M records | ~60–70 MB MMDB | 69 MB BIN (IPv4) + 171 MB BIN (IPv6), separate files |
|
|||
|
|
| Format / reader | **MMDB → `maxminddb`** (pure-Python, mmap) | MMDB → `maxminddb` / `geoip2` | proprietary BIN → vendor `IP2Location` package (or CSV you index yourself) |
|
|||
|
|
| Accuracy fields | city + lat/lon, **no `accuracy_radius`** | city + lat/lon + `accuracy_radius` | city + lat/lon |
|
|||
|
|
|
|||
|
|
**Recommend DB-IP IP to City Lite.** Reasoning, in priority order:
|
|||
|
|
|
|||
|
|
1. **No credential.** MaxMind means a licence key in the SOPS vault, in the
|
|||
|
|
deploy path, and in a rotation story — new secret surface for a feature whose
|
|||
|
|
entire justification is privacy hygiene. DB-IP needs none, so
|
|||
|
|
`geoip-refresh.sh` reads nothing from the vault at all.
|
|||
|
|
2. **No retention obligation.** GeoLite's EULA requires deleting a database
|
|||
|
|
within 30 days of a newer release. That is a *compliance* clock on an ops
|
|||
|
|
job, and a missed job becomes a licence breach rather than merely stale data.
|
|||
|
|
CC BY 4.0 has no such clause.
|
|||
|
|
3. **Same format, same reader.** Both are MMDB, so this is a swap of one file
|
|||
|
|
and one URL, not a code change — `geoip.py` reads the raw record dict and
|
|||
|
|
derives its attribution line from the file's own metadata. If accuracy turns
|
|||
|
|
out to be materially worse in practice, moving to GeoLite2 is a config change
|
|||
|
|
plus a vault entry, and the code is already written to accept it.
|
|||
|
|
4. **Attribution is a link, not a legal paragraph.** One muted anchor in the
|
|||
|
|
suggestion banner satisfies CC BY 4.0 and DB-IP's stated wording.
|
|||
|
|
|
|||
|
|
**Against IP2Location LITE**: a proprietary BIN format needing the vendor's own
|
|||
|
|
Python package, IPv4 and IPv6 as *separate* databases (two files, two readers,
|
|||
|
|
two refresh paths), an account to download, and CC BY-**SA** — a share-alike
|
|||
|
|
term that is noise to reason about for a file we merely read.
|
|||
|
|
|
|||
|
|
**Cost of the DB-IP choice, stated plainly:** the Lite file carries no
|
|||
|
|
`accuracy_radius`, so the "reject anything the database admits is too coarse"
|
|||
|
|
guard degrades to "reject anything with no city name". That is still the guard
|
|||
|
|
that matters (a country centroid is the damaging case), but it is weaker.
|
|||
|
|
`suggestion_from_record()` applies the radius rule whenever the field *is*
|
|||
|
|
present, so switching to GeoLite2 turns the stronger guard back on for free.
|
|||
|
|
|
|||
|
|
**Reader library:** `maxminddb` (pinned `2.8.2` in `backend/requirements.txt`),
|
|||
|
|
not `geoip2`. `geoip2` adds a model layer over the same reader and type-checks
|
|||
|
|
`database_type` against MaxMind's own names; we read four fields out of a plain
|
|||
|
|
dict, and staying on `maxminddb` keeps DB-IP and MaxMind files equally usable.
|
|||
|
|
It is imported *lazily inside* `geoip.py`, so a missing wheel disables the
|
|||
|
|
feature rather than breaking boot.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 4. Where it hooks in — recommendation: **client-side, on demand**
|
|||
|
|
|
|||
|
|
Two options were on the table.
|
|||
|
|
|
|||
|
|
**A. Server-side during SSR of `/`.** `frontend/content.py`'s `home_page()`
|
|||
|
|
would look up the IP and render the guess into the first paint.
|
|||
|
|
|
|||
|
|
**B. Client-side after geolocation is declined/unavailable.** The browser calls
|
|||
|
|
`GET /api/v2/geoip` and swaps the banner in. **This is what the prototype does.**
|
|||
|
|
|
|||
|
|
### Why B
|
|||
|
|
|
|||
|
|
**Caching is the deciding factor, and the current setup is more fragile than it
|
|||
|
|
looks.** `frontend/content.py`'s `_respond_html()` returns HTML with a weak
|
|||
|
|
ETag (`W/"<sha1 of the rendered body>"`) and **no `Cache-Control` and no
|
|||
|
|
`Vary`**. The host Caddy (`infra/deploy/Caddyfile`) only does `encode` +
|
|||
|
|
`reverse_proxy` — no cache module today. So right now nothing caches the
|
|||
|
|
homepage, but nothing *declares* that it mustn't either.
|
|||
|
|
|
|||
|
|
Fold a location into that response and three things break at once:
|
|||
|
|
|
|||
|
|
- The ETag becomes a function of the visitor's city. Two visitors flap each
|
|||
|
|
other's validators; conditional requests stop being useful, and a returning
|
|||
|
|
visitor whose IP maps elsewhere gets a spurious 200 instead of a 304.
|
|||
|
|
- Any cache added later — a Caddy `cache` directive, Cloudflare in front of the
|
|||
|
|
apex, a corporate proxy on the visitor's side — will serve one visitor's city
|
|||
|
|
to another, because the response carries nothing saying it varies. There is no
|
|||
|
|
correct `Vary` header for this: it varies on `X-Forwarded-For`, which is not a
|
|||
|
|
header a cache keys on, and `Vary: *` would make the homepage permanently
|
|||
|
|
uncacheable for everyone to serve a fallback that only a minority need.
|
|||
|
|
- The SSR path would also have to carry the client IP over the extra
|
|||
|
|
frontend→backend hop (`api_client.py` currently forwards only `Host` and
|
|||
|
|
`X-Forwarded-Proto`). That means propagating the IP into a second process's
|
|||
|
|
request path — more places it can end up in a log, for no benefit.
|
|||
|
|
|
|||
|
|
**The first-paint argument is weaker here than it looks**, because the homepage
|
|||
|
|
hero is *already* server-rendered with real content: the most unusual city
|
|||
|
|
we're currently tracking, with its grade, band colour and percentile. There is
|
|||
|
|
no blank space for the guess to fill. The suggestion swaps into a page that is
|
|||
|
|
already complete, and it costs one small JSON request with no data dependency in
|
|||
|
|
front of it.
|
|||
|
|
|
|||
|
|
**And SSR would do the lookup for the wrong people.** The server cannot see
|
|||
|
|
`localStorage`, so it cannot know that most returning visitors already have a
|
|||
|
|
saved place that is strictly better than any guess. An SSR implementation would
|
|||
|
|
geolocate every homepage request, including the majority who will never use the
|
|||
|
|
result — the opposite of "touch the IP as little as possible". Option B only
|
|||
|
|
runs after the visitor has demonstrably ended up with nothing.
|
|||
|
|
|
|||
|
|
### What B costs
|
|||
|
|
|
|||
|
|
- **Needs JavaScript.** A no-JS visitor sees today's behaviour. Acceptable: the
|
|||
|
|
entire interactive tool already needs JS.
|
|||
|
|
- **One extra round trip** after the decline, ~200 bytes each way.
|
|||
|
|
- **A 204 for everyone in the fallback path when the feature is off.** The
|
|||
|
|
alternative — giving the frontend its own copy of the flag to decide whether
|
|||
|
|
to fetch — is two flags to keep in sync and a new way for the UI and the
|
|||
|
|
server to disagree. One flag, one 204, is the better trade.
|
|||
|
|
|
|||
|
|
### If you overrule this and want SSR anyway
|
|||
|
|
|
|||
|
|
Do it as a *separate* route (`/near` or `?near=1`), not on the cached `/`, and
|
|||
|
|
give that route `Cache-Control: private, no-store`. Never make the canonical,
|
|||
|
|
indexable homepage location-varying — that is also an SEO hazard: a crawler in
|
|||
|
|
one datacentre would index a page claiming a city, and different crawler IPs
|
|||
|
|
would see different "canonical" content.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 5. Privacy design
|
|||
|
|
|
|||
|
|
### The IP is never persisted — mechanically, not by convention
|
|||
|
|
|
|||
|
|
- `geoip.lookup(ip)` takes a string, memory-maps a file, returns a dict. It
|
|||
|
|
writes nothing, logs nothing, and never raises.
|
|||
|
|
- `api_geoip` uses **no `RunAudit`** (which would record the derived lat/lon)
|
|||
|
|
and adds **no metric dimension** (the counters are aggregate by category).
|
|||
|
|
- The route is excluded from the access log. `metrics.classify_inbound()` now
|
|||
|
|
returns a dedicated `"geoip"` category and the middleware in `web/app.py`
|
|||
|
|
skips `audit.log_access` for it. Every other route still logs, and
|
|||
|
|
`test_a_normal_route_still_is_logged` guards that exclusion from widening.
|
|||
|
|
|
|||
|
|
This is the important one. Today the access log stores a raw IP per request.
|
|||
|
|
Without the exclusion, the *one* endpoint whose whole job is turning an IP
|
|||
|
|
into a place would leave a stored, timestamped, joinable (IP,
|
|||
|
|
approximate-location) pair on disk and in Loki. That artefact — not the
|
|||
|
|
transient lookup — is what would actually constitute keeping location data
|
|||
|
|
about identifiable people. The exclusion is not defence in depth; it is the
|
|||
|
|
substance of the claim.
|
|||
|
|
|
|||
|
|
This does **not** depend on the separate access-log IP-truncation work. That
|
|||
|
|
change is still worth doing, and it composes: truncation reduces what the
|
|||
|
|
*other* routes retain, while this exclusion means the geoip route retains
|
|||
|
|
nothing at all either way.
|
|||
|
|
|
|||
|
|
- `Cache-Control: no-store, private` + `Vary: *` on the response, so it cannot
|
|||
|
|
be stored or reused by any intermediary.
|
|||
|
|
- The endpoint takes **no parameters**. You cannot ask it about somebody else's
|
|||
|
|
address; the only input is the connection you already own. `?ip=` is ignored
|
|||
|
|
(tested).
|
|||
|
|
|
|||
|
|
### Why no consent banner is needed — and why the analytics work is different
|
|||
|
|
|
|||
|
|
Under GDPR the lookup is **transient processing to deliver a service the user
|
|||
|
|
explicitly asked for**. The visitor pressed "Use my location" *wanting* local
|
|||
|
|
weather; when the browser can't supply it, deriving a coarse city from the
|
|||
|
|
connection they are already using to reach us is a proportionate substitute for
|
|||
|
|
the same, requested purpose. Nothing is stored, nothing is profiled, nothing is
|
|||
|
|
shared with a third party (the database is a file we host — no processor
|
|||
|
|
relationship, no international transfer, no disclosure to name in the policy).
|
|||
|
|
The data subject can override the result with one tap, and the result is
|
|||
|
|
explicitly labelled a guess rather than presented as a finding about them.
|
|||
|
|
|
|||
|
|
**ePrivacy Art. 5(3) is not engaged at all**, and this is the cleanest part of
|
|||
|
|
the argument. That article governs *storing information on, or gaining access to
|
|||
|
|
information stored in, a user's terminal equipment* — which is why cookies,
|
|||
|
|
localStorage and device fingerprinting need consent regardless of whether the
|
|||
|
|
data is personal. An IP address arrives in the request headers as an unavoidable
|
|||
|
|
property of routing a packet. We neither store anything on the device nor read
|
|||
|
|
anything from it. There is nothing for a consent banner to be about.
|
|||
|
|
|
|||
|
|
**Contrast with the analytics session-id work.** That one deliberately *writes
|
|||
|
|
an identifier onto the visitor's device* in order to correlate their actions
|
|||
|
|
across a session. That is squarely Art. 5(3) storage/access, it is not strictly
|
|||
|
|
necessary to deliver a service the user requested, and consent is therefore
|
|||
|
|
required — hence `consent.js` and the consent gate on that branch. The two
|
|||
|
|
features look adjacent ("both touch visitor data") and are legally opposite:
|
|||
|
|
one reads a routing header and forgets it, the other plants a durable
|
|||
|
|
identifier. Keep them separate in the code and in the privacy copy, and never
|
|||
|
|
let the geoip fallback be folded behind the analytics consent gate — that would
|
|||
|
|
imply consent is what makes it lawful, which is the wrong basis and would
|
|||
|
|
silently break the fallback for everyone who declines analytics.
|
|||
|
|
|
|||
|
|
### The privacy page follows the flag
|
|||
|
|
|
|||
|
|
`frontend/templates/privacy.html.j2` currently says, in the visitor's own
|
|||
|
|
words: *"Thermograph never looks up your location from your IP address."*
|
|||
|
|
Turning this feature on makes that sentence **false**, and a stale privacy
|
|||
|
|
statement is a worse outcome than not shipping the feature.
|
|||
|
|
|
|||
|
|
So the paragraph is switched on `THERMOGRAPH_GEOIP`, which both containers read
|
|||
|
|
from the same `/etc/thermograph.env` (`content.py`'s `geoip_enabled()`). The
|
|||
|
|
statement and the behaviour flip in the same deploy, and the frontend unit tests
|
|||
|
|
assert both branches. The frontend reads the flag for *nothing else* — it does
|
|||
|
|
not gate any behaviour on it.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 6. Graceful degradation
|
|||
|
|
|
|||
|
|
Every one of these returns `None` → HTTP 204 → the client does exactly what the
|
|||
|
|
site does today. There is no failure mode worse than the status quo.
|
|||
|
|
|
|||
|
|
| Case | Handled where | Behaviour |
|
|||
|
|
| --- | --- | --- |
|
|||
|
|
| Flag unset | `enabled()` | 204 |
|
|||
|
|
| Flag set, no database file | `enabled()` (`os.path.isfile`) | 204 |
|
|||
|
|
| `maxminddb` not installed | lazy import in `_open()` | 204 |
|
|||
|
|
| Truncated / corrupt / half-written file | `_open()` try/except | 204 (tested) |
|
|||
|
|
| Private ranges (10/8, 172.16/12, 192.168/16) | `parse_ip` | 204 — every LAN-dev request |
|
|||
|
|
| Loopback, link-local, multicast, reserved, unspecified | `parse_ip` | 204 |
|
|||
|
|
| **CGNAT 100.64/10** | `parse_ip` + explicit `_EXTRA_RESERVED` | 204 — common on mobile carriers |
|
|||
|
|
| IPv6 (global) | `parse_ip` | looked up normally; MMDB covers v6 |
|
|||
|
|
| IPv6 ULA `fc00::/7`, doc `2001:db8::/32` | `_EXTRA_RESERVED` | 204 |
|
|||
|
|
| IPv4-mapped IPv6 `::ffff:a.b.c.d` | `parse_ip` unwraps | treated as the v4 address |
|
|||
|
|
| `1.2.3.4:5678`, `[v6]:443`, `%zone` suffixes | `parse_ip` | normalised |
|
|||
|
|
| Missing `X-Forwarded-For` | `_client_ip` falls back to peer | peer is loopback behind Caddy → 204 |
|
|||
|
|
| A whole XFF chain passed as one string | `parse_ip` | rejected (the route already takes the left-most hop) |
|
|||
|
|
| No record for the address | `lookup()` | 204 |
|
|||
|
|
| Record with a country but **no city** | `suggestion_from_record` | 204 — the VPN/proxy case |
|
|||
|
|
| Record whose stated radius > 100 km | `suggestion_from_record` | 204 |
|
|||
|
|
| Reader raises on a bad node | `lookup()` try/except | 204 |
|
|||
|
|
|
|||
|
|
The explicit `_EXTRA_RESERVED` list is not redundant with `ipaddress.is_global`:
|
|||
|
|
the exact membership of `is_private`/`is_global` has moved between CPython
|
|||
|
|
releases (100.64/10 in particular). Which Python the container happens to run
|
|||
|
|
must not decide whether we try to geolocate a carrier-NAT address.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 7. The UI — presenting a guess as a guess
|
|||
|
|
|
|||
|
|
Copy shipped in `frontend/static/geoip.js`:
|
|||
|
|
|
|||
|
|
> **Showing weather near Rotterdam, South Holland.**
|
|||
|
|
> That is a rough guess from your internet connection, not from your device — it
|
|||
|
|
> can be tens of kilometres out. *IP Geolocation by DB-IP*
|
|||
|
|
>
|
|||
|
|
> **[ Not right? Choose your spot ]**
|
|||
|
|
|
|||
|
|
Design rules, and why each one:
|
|||
|
|
|
|||
|
|
1. **Named as a guess, in the same breath as the result.** Not a footnote, not a
|
|||
|
|
tooltip. City-level accuracy is ~50 km against a ~2 mile grid cell — the
|
|||
|
|
product's own unit of precision is 25× finer than the input.
|
|||
|
|
2. **The correction is a permanent, full-width, ≥44 px control**, not a
|
|||
|
|
dismissible toast. If the guess is wrong, fixing it must be the most obvious
|
|||
|
|
thing on screen.
|
|||
|
|
3. **A guess is never persisted.** `updateHash()` returns early while
|
|||
|
|
`approx` is true: no `localStorage` write, no URL hash. A guess that became
|
|||
|
|
sticky would be indistinguishable from a choice on the next visit, and a
|
|||
|
|
guessed URL would propagate a rough location the sender never picked when
|
|||
|
|
shared. Only an explicit pick is remembered.
|
|||
|
|
4. **The headings are rewritten while approximate.** The hero says "Roughly near
|
|||
|
|
Rotterdam" instead of "Today in …", and the results heading says "Near
|
|||
|
|
Rotterdam" with an "approximate — from your connection, not your device"
|
|||
|
|
tag. This matters more than it sounds: the guessed cell centre gets
|
|||
|
|
reverse-geocoded like any other point, which would otherwise name a
|
|||
|
|
*neighbourhood* — presenting street-level precision for a city-level input.
|
|||
|
|
5. **Visually recessed** — `--surface-2`, dashed border, muted note text. It
|
|||
|
|
reads as an aside, not as an answer.
|
|||
|
|
6. **It never pre-empts the permission prompt.** It runs *after* a
|
|||
|
|
declined/failed prompt, or on load only where the browser will not offer
|
|||
|
|
geolocation at all (insecure context) and nothing is remembered.
|
|||
|
|
7. **Never for a visitor who already has a place.** Theirs is better.
|
|||
|
|
|
|||
|
|
Attribution rides *with the result*, which is what CC BY 4.0 and DB-IP's own
|
|||
|
|
wording ask for ("on pages displaying results") — and it is derived from the
|
|||
|
|
installed file's metadata, so swapping databases changes the credit
|
|||
|
|
automatically.
|
|||
|
|
|
|||
|
|
### Interaction with `home.locate`
|
|||
|
|
|
|||
|
|
`home.locate` still means exactly what it means today: "someone tapped *Use my
|
|||
|
|
location*", whatever the outcome. It is unchanged, so the existing series stays
|
|||
|
|
comparable across this change. Two new allowlisted events sit beside it:
|
|||
|
|
|
|||
|
|
- `home.geoip_shown` — a suggestion was rendered. `home.locate` minus
|
|||
|
|
`home.geoip_shown` is roughly the population still hitting the dead end.
|
|||
|
|
- `home.geoip_fixed` — the visitor corrected the guess. This is the accuracy
|
|||
|
|
signal: a high ratio means the database is putting people in the wrong place
|
|||
|
|
and the feature is doing harm, which is the number to watch after rollout.
|
|||
|
|
|
|||
|
|
Both are aggregate counts through the existing beacon, with **no location
|
|||
|
|
dimension** — the suggestion itself is never recorded anywhere.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 8. Database lifecycle
|
|||
|
|
|
|||
|
|
**On the host, refreshed by a systemd timer, bind-mounted read-only.** Not baked
|
|||
|
|
into the image.
|
|||
|
|
|
|||
|
|
- `infra/deploy/geoip-refresh.sh` — resolves the current month's DB-IP release
|
|||
|
|
(falling back to last month's, since the new file appears partway through the
|
|||
|
|
1st), downloads, gunzips, **verifies it opens and answers for a known public
|
|||
|
|
address**, and only then swaps it in atomically with `install`+`mv`. Exits 0
|
|||
|
|
on "already current". A failure leaves the existing file untouched — a failed
|
|||
|
|
refresh must never be worse than a stale one.
|
|||
|
|
- `geoip-refresh.timer` — `OnCalendar=Mon 04:17 UTC`, `RandomizedDelaySec=2h`,
|
|||
|
|
`Persistent=true`. Weekly rather than monthly even though DB-IP publishes
|
|||
|
|
monthly: a no-op run costs one HEAD request, and it means a missed or failed
|
|||
|
|
run self-heals within a week. (It also lands inside MaxMind's 30-day deletion
|
|||
|
|
window without further thought, should you switch.)
|
|||
|
|
- `infra/docker-compose.yml` mounts `${THERMOGRAPH_GEOIP_HOST_DIR:-/var/lib/thermograph/geoip}:/geoip:ro`
|
|||
|
|
into the backend. Read-only on purpose — the app must never be able to write it.
|
|||
|
|
- `geoip.py` re-opens when the file's `(mtime, size)` changes, so a refresh is
|
|||
|
|
picked up **without a restart or a deploy**.
|
|||
|
|
- `MODE_MMAP`: the 125 MB file costs page cache, not process RSS, and is shared
|
|||
|
|
across the four uvicorn workers on a host.
|
|||
|
|
|
|||
|
|
**Why not build-time.** It is a 125 MB data file on its own release cadence, not
|
|||
|
|
code. Baking it in adds 125 MB to every image pull, freezes freshness to
|
|||
|
|
whenever the last deploy happened, ties a data refresh to a code deploy, and
|
|||
|
|
(under GeoLite's EULA) cannot honour the 30-day deletion requirement on its own.
|
|||
|
|
|
|||
|
|
**Why not object storage.** We already have the bucket and could pull from it,
|
|||
|
|
but that adds credentials and a network dependency to fetch a file that is
|
|||
|
|
public and unauthenticated at the source. If a mirror is wanted later — to stop
|
|||
|
|
depending on DB-IP's uptime, or to pin exactly which release is live across the
|
|||
|
|
fleet — point `GEOIP_URL` at the bucket; the script needs no other change.
|
|||
|
|
|
|||
|
|
**Licence key storage.** With DB-IP: **none needed**, which is a large part of
|
|||
|
|
the recommendation. If MaxMind is chosen instead, the key goes in the SOPS vault
|
|||
|
|
(`infra/deploy/secrets/{prod,beta}.yaml`), renders into `/etc/thermograph.env`
|
|||
|
|
like every other secret, and `geoip-refresh.service` already picks it up via its
|
|||
|
|
`EnvironmentFile=` — set `GEOIP_URL` to the permalink that embeds it. Note the
|
|||
|
|
key would then be in a *URL*, so keep it out of any log line that echoes the
|
|||
|
|
URL: `geoip-refresh.sh` logs `$url`, and that line must be changed before using
|
|||
|
|
a key-bearing URL. **Flagged as a real footgun if you pick MaxMind.**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 9. Interaction with caching / CDN
|
|||
|
|
|
|||
|
|
- **The homepage stays byte-identical for every visitor** — the container is
|
|||
|
|
server-rendered empty and hidden. `test_home_html_is_identical_regardless_of_client_address`
|
|||
|
|
asserts the body *and* the ETag match across different `X-Forwarded-For`
|
|||
|
|
values. That property is what keeps the homepage safe to cache, and it is the
|
|||
|
|
main reason the lookup is not in SSR.
|
|||
|
|
- **The endpoint is `no-store, private` + `Vary: *`** and answers 204 far more
|
|||
|
|
often than 200. Nothing about it is shareable between visitors.
|
|||
|
|
- **A CDN in front of the apex stays viable.** If Cloudflare (or a Caddy `cache`
|
|||
|
|
directive) is added later, the rule is one line: never cache `/api/*`. That is
|
|||
|
|
already the sane default and already how Caddy path-splits.
|
|||
|
|
- **The existing frontend caches are untouched.** `api_client.py`'s TTL/LRU
|
|||
|
|
cache never sees this request (it is browser→backend, not frontend→backend),
|
|||
|
|
and `static/cache.js`'s IndexedDB layer is keyed by `/cell` URLs. A guessed
|
|||
|
|
location produces ordinary `/api/v2/grade` traffic for an ordinary cell, so it
|
|||
|
|
warms and reuses the same caches as any other place — no new cache keyspace.
|
|||
|
|
- **One second-order effect worth naming:** guessed locations concentrate on
|
|||
|
|
city centroids, so they will warm the *same* handful of cells repeatedly.
|
|||
|
|
That is good for us (high cache hit rate, no extra Open-Meteo quota) but it
|
|||
|
|
does mean per-cell request counts stop being a clean proxy for distinct users
|
|||
|
|
near that cell.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 10. Rollout
|
|||
|
|
|
|||
|
|
1. Merge with the flag off. Nothing changes — that is the point, and the tests
|
|||
|
|
assert it.
|
|||
|
|
2. On **beta only**: install the timer, run `geoip-refresh.sh` once by hand,
|
|||
|
|
confirm `/var/lib/thermograph/geoip/city.mmdb` exists and
|
|||
|
|
`curl -s -o /dev/null -w '%{http_code}' https://beta.thermograph.org/api/v2/geoip`
|
|||
|
|
still returns 204 (flag still off).
|
|||
|
|
3. Set `THERMOGRAPH_GEOIP=1` in beta's vault entry, redeploy **both** containers
|
|||
|
|
(backend for the lookup, frontend for the privacy copy).
|
|||
|
|
4. Verify by hand: decline the permission prompt; confirm the banner appears,
|
|||
|
|
the correction button works, the guess is *not* in `localStorage` and *not*
|
|||
|
|
in the URL; confirm `/privacy` now describes the fallback; confirm no
|
|||
|
|
`"cat":"geoip"` line and no matching IP appears in the access log.
|
|||
|
|
5. Watch `home.geoip_fixed / home.geoip_shown` for a week. If a large share of
|
|||
|
|
visitors are correcting the guess, the database is putting people in the
|
|||
|
|
wrong place — turn it off rather than tuning copy.
|
|||
|
|
6. Only then, prod.
|
|||
|
|
|
|||
|
|
Rollback is `THERMOGRAPH_GEOIP=0` and a redeploy; or just delete the database
|
|||
|
|
file, which disables it without touching config.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 11. Decisions needed from the operator
|
|||
|
|
|
|||
|
|
1. **Which database?** Recommendation: **DB-IP IP to City Lite** (no account, no
|
|||
|
|
key, no retention clock, CC BY 4.0). The alternative is GeoLite2 City for
|
|||
|
|
`accuracy_radius` and weekly updates, at the cost of a vault-managed licence
|
|||
|
|
key and a 30-day deletion obligation. Code supports either unchanged.
|
|||
|
|
2. **Attribution placement.** Currently in the suggestion banner itself, beside
|
|||
|
|
the result. Alternatives: also on `/about`, or only on `/about`. DB-IP's
|
|||
|
|
stated wording asks for it "on pages displaying results", which the banner
|
|||
|
|
satisfies — confirm you're happy with a small credit line inside the hero.
|
|||
|
|
3. **SSR or client-side?** Recommendation: **client-side** (§4). Overruling this
|
|||
|
|
means accepting a location-varying homepage; if you do, put it on a separate
|
|||
|
|
non-canonical route, never on cached `/`.
|
|||
|
|
4. **Refresh cadence.** Currently weekly-with-no-op (§8). Alternatives: monthly
|
|||
|
|
to match DB-IP's release cadence exactly, or daily. Weekly is the
|
|||
|
|
self-healing middle.
|
|||
|
|
5. **Auto-suggest in insecure contexts?** Today the prototype suggests on load
|
|||
|
|
when the browser won't offer geolocation at all (LAN dev over plain HTTP) and
|
|||
|
|
nothing is remembered, since there is no prompt to decline. That is the only
|
|||
|
|
path where a lookup happens without an explicit user action. Prod is HTTPS so
|
|||
|
|
it never triggers there — but say if you'd rather it *never* happen without a
|
|||
|
|
tap, and it becomes a one-line change.
|
|||
|
|
6. **Threshold for calling it off.** Pick the `home.geoip_fixed / home.geoip_shown`
|
|||
|
|
ratio at which the feature gets turned off, *before* rollout rather than
|
|||
|
|
after — otherwise the number will always look explainable.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 12. Things I think are bad ideas — flagged, not implemented
|
|||
|
|
|
|||
|
|
- **Don't put the lookup in the SSR homepage.** §4. A location-varying
|
|||
|
|
canonical page is a cache-poisoning and SEO hazard, and it geolocates the
|
|||
|
|
majority of visitors who already have a saved place and will never use it.
|
|||
|
|
- **Don't persist the guess.** Not to `localStorage`, not to the URL, not to an
|
|||
|
|
account preference. It is not a choice and must never become one silently.
|
|||
|
|
The branch enforces this in `updateHash()`.
|
|||
|
|
- **Don't use a third-party geolocation API**, even a "privacy-friendly" one.
|
|||
|
|
Sending visitor IPs to another company creates a processor relationship and a
|
|||
|
|
disclosure obligation, for a file we can just host. This was already the
|
|||
|
|
brief's requirement; restating it because it is the single most tempting
|
|||
|
|
shortcut here.
|
|||
|
|
- **Don't extend this to country-level personalisation** (units, language,
|
|||
|
|
default city) on the same lookup. Unit selection already works off the picked
|
|||
|
|
location's country code, and quietly widening an IP lookup from "a fallback
|
|||
|
|
the user asked for" to "we adapt the site to you" changes the legal analysis
|
|||
|
|
in §5 — it stops being strictly necessary for a requested service.
|
|||
|
|
- **Don't reuse the derived city anywhere it could be stored.** No metrics
|
|||
|
|
dimension, no audit field, no "popular cities" table fed from guesses. The
|
|||
|
|
moment a derived location is written next to anything timestamped, the
|
|||
|
|
privacy claim in §5 stops being true.
|
|||
|
|
- **Don't fold this behind the analytics consent gate.** §5. Different legal
|
|||
|
|
basis; gating it on consent both implies the wrong basis and breaks the
|
|||
|
|
fallback for everyone who declines analytics.
|
|||
|
|
- **Don't log `$url` once a MaxMind key-bearing permalink is in use** (§8).
|
|||
|
|
- **Be honest that this is a downgrade, not a feature.** A city-level guess is
|
|||
|
|
25× coarser than the grid the whole product is built on. It exists so a
|
|||
|
|
visitor who declined isn't stranded — not because IP location is good. If the
|
|||
|
|
correction rate says people don't want it, delete it rather than defend it.
|