# Approximate-location fallback — design + prototype Status: **prototype on `feat/geoip-approx-location`, feature-flagged OFF, not deployed and not merged.** Nothing in this branch changes production behaviour until `THERMOGRAPH_GEOIP` is set *and* a database file exists on the host. This doc is the decision record. If it survives review it should move to `thermograph-docs` (per the root `CLAUDE.md`: cross-cutting decision docs live there); it sits here for now so the reasoning ships with the code it describes. --- ## 1. The problem Location today comes from exactly one source: `navigator.geolocation.getCurrentPosition` in `frontend/static/app.js`, gated on `window.isSecureContext`. If the visitor declines the permission prompt, the prompt times out, or the browser won't offer geolocation at all, the site has **no location** and the copy says "Pick a spot on the map instead." That is a dead end for the exact visitor the homepage is written for — someone who arrived to answer "how unusual is *my* weather?" and has just been handed a world map. This change gives that person something local, honestly labelled. ## 2. What was built | Piece | File | | --- | --- | | Lookup module (flag, parsing, rejection rules, attribution) | `backend/data/geoip.py` | | Endpoint `GET /api/v2/geoip` | `backend/web/app.py` (`api_geoip`) | | Access-log exclusion + traffic category | `backend/web/app.py`, `backend/core/metrics.py` | | Event names `home.geoip_shown` / `home.geoip_fixed` | `backend/core/metrics.py` | | Client fetch + suggestion banner | `frontend/static/geoip.js` | | Wiring, non-persistence, honest headings | `frontend/static/app.js` | | Banner container (empty, hidden) | `frontend/templates/home.html.j2` | | Banner styles | `frontend/static/style.css` | | Flag-aware privacy copy | `frontend/content.py`, `frontend/templates/privacy.html.j2` | | Host refresh job + timer | `infra/deploy/geoip-refresh.{sh,service,timer}` | | Read-only DB mount + env plumbing | `infra/docker-compose.yml`, `infra/deploy/thermograph.env.example` | | Tests | `backend/tests/data/test_geoip.py`, `backend/tests/web/test_geoip_api.py`, `frontend/tests/unit/test_geoip_ui.py` | `cd backend && make test` → **449 passed, 8 skipped** (was 386/7). `cd frontend && make test-unit` → **41 passed**. --- ## 3. Database choice — recommendation: **DB-IP IP to City Lite** | | **DB-IP IP to City Lite** | MaxMind GeoLite2 City | IP2Location LITE DB5/DB11 | | --- | --- | --- | --- | | Licence | **CC BY 4.0** | GeoLite End User License Agreement (incorporates CC BY-SA 4.0 terms) | CC BY-SA 4.0 / "free with attribution" | | Account needed | **No** — direct download URL | **Yes** — account + licence key | **Yes** — free account | | Secret to manage | **None** | Licence key (→ SOPS vault, → CI, → rotation) | Account credentials | | Attribution | `IP Geolocation by DB-IP` on pages showing results | "This product includes GeoLite Data created by MaxMind, available from https://www.maxmind.com" | "uses the IP2Location LITE database for IP geolocation", linked | | Retention obligation | none | **must delete a database within 30 days of a newer release**; 30 downloads/day cap | none | | Update cadence | monthly | weekly (Tue/Fri) | monthly | | Size (current release) | 125 MB MMDB (685 MB CSV), ~8.05 M records | ~60–70 MB MMDB | 69 MB BIN (IPv4) + 171 MB BIN (IPv6), separate files | | Format / reader | **MMDB → `maxminddb`** (pure-Python, mmap) | MMDB → `maxminddb` / `geoip2` | proprietary BIN → vendor `IP2Location` package (or CSV you index yourself) | | Accuracy fields | city + lat/lon, **no `accuracy_radius`** | city + lat/lon + `accuracy_radius` | city + lat/lon | **Recommend DB-IP IP to City Lite.** Reasoning, in priority order: 1. **No credential.** MaxMind means a licence key in the SOPS vault, in the deploy path, and in a rotation story — new secret surface for a feature whose entire justification is privacy hygiene. DB-IP needs none, so `geoip-refresh.sh` reads nothing from the vault at all. 2. **No retention obligation.** GeoLite's EULA requires deleting a database within 30 days of a newer release. That is a *compliance* clock on an ops job, and a missed job becomes a licence breach rather than merely stale data. CC BY 4.0 has no such clause. 3. **Same format, same reader.** Both are MMDB, so this is a swap of one file and one URL, not a code change — `geoip.py` reads the raw record dict and derives its attribution line from the file's own metadata. If accuracy turns out to be materially worse in practice, moving to GeoLite2 is a config change plus a vault entry, and the code is already written to accept it. 4. **Attribution is a link, not a legal paragraph.** One muted anchor in the suggestion banner satisfies CC BY 4.0 and DB-IP's stated wording. **Against IP2Location LITE**: a proprietary BIN format needing the vendor's own Python package, IPv4 and IPv6 as *separate* databases (two files, two readers, two refresh paths), an account to download, and CC BY-**SA** — a share-alike term that is noise to reason about for a file we merely read. **Cost of the DB-IP choice, stated plainly:** the Lite file carries no `accuracy_radius`, so the "reject anything the database admits is too coarse" guard degrades to "reject anything with no city name". That is still the guard that matters (a country centroid is the damaging case), but it is weaker. `suggestion_from_record()` applies the radius rule whenever the field *is* present, so switching to GeoLite2 turns the stronger guard back on for free. **Reader library:** `maxminddb` (pinned `2.8.2` in `backend/requirements.txt`), not `geoip2`. `geoip2` adds a model layer over the same reader and type-checks `database_type` against MaxMind's own names; we read four fields out of a plain dict, and staying on `maxminddb` keeps DB-IP and MaxMind files equally usable. It is imported *lazily inside* `geoip.py`, so a missing wheel disables the feature rather than breaking boot. --- ## 4. Where it hooks in — recommendation: **client-side, on demand** Two options were on the table. **A. Server-side during SSR of `/`.** `frontend/content.py`'s `home_page()` would look up the IP and render the guess into the first paint. **B. Client-side after geolocation is declined/unavailable.** The browser calls `GET /api/v2/geoip` and swaps the banner in. **This is what the prototype does.** ### Why B **Caching is the deciding factor, and the current setup is more fragile than it looks.** `frontend/content.py`'s `_respond_html()` returns HTML with a weak ETag (`W/""`) and **no `Cache-Control` and no `Vary`**. The host Caddy (`infra/deploy/Caddyfile`) only does `encode` + `reverse_proxy` — no cache module today. So right now nothing caches the homepage, but nothing *declares* that it mustn't either. Fold a location into that response and three things break at once: - The ETag becomes a function of the visitor's city. Two visitors flap each other's validators; conditional requests stop being useful, and a returning visitor whose IP maps elsewhere gets a spurious 200 instead of a 304. - Any cache added later — a Caddy `cache` directive, Cloudflare in front of the apex, a corporate proxy on the visitor's side — will serve one visitor's city to another, because the response carries nothing saying it varies. There is no correct `Vary` header for this: it varies on `X-Forwarded-For`, which is not a header a cache keys on, and `Vary: *` would make the homepage permanently uncacheable for everyone to serve a fallback that only a minority need. - The SSR path would also have to carry the client IP over the extra frontend→backend hop (`api_client.py` currently forwards only `Host` and `X-Forwarded-Proto`). That means propagating the IP into a second process's request path — more places it can end up in a log, for no benefit. **The first-paint argument is weaker here than it looks**, because the homepage hero is *already* server-rendered with real content: the most unusual city we're currently tracking, with its grade, band colour and percentile. There is no blank space for the guess to fill. The suggestion swaps into a page that is already complete, and it costs one small JSON request with no data dependency in front of it. **And SSR would do the lookup for the wrong people.** The server cannot see `localStorage`, so it cannot know that most returning visitors already have a saved place that is strictly better than any guess. An SSR implementation would geolocate every homepage request, including the majority who will never use the result — the opposite of "touch the IP as little as possible". Option B only runs after the visitor has demonstrably ended up with nothing. ### What B costs - **Needs JavaScript.** A no-JS visitor sees today's behaviour. Acceptable: the entire interactive tool already needs JS. - **One extra round trip** after the decline, ~200 bytes each way. - **A 204 for everyone in the fallback path when the feature is off.** The alternative — giving the frontend its own copy of the flag to decide whether to fetch — is two flags to keep in sync and a new way for the UI and the server to disagree. One flag, one 204, is the better trade. ### If you overrule this and want SSR anyway Do it as a *separate* route (`/near` or `?near=1`), not on the cached `/`, and give that route `Cache-Control: private, no-store`. Never make the canonical, indexable homepage location-varying — that is also an SEO hazard: a crawler in one datacentre would index a page claiming a city, and different crawler IPs would see different "canonical" content. --- ## 5. Privacy design ### The IP is never persisted — mechanically, not by convention - `geoip.lookup(ip)` takes a string, memory-maps a file, returns a dict. It writes nothing, logs nothing, and never raises. - `api_geoip` uses **no `RunAudit`** (which would record the derived lat/lon) and adds **no metric dimension** (the counters are aggregate by category). - The route is excluded from the access log. `metrics.classify_inbound()` now returns a dedicated `"geoip"` category and the middleware in `web/app.py` skips `audit.log_access` for it. Every other route still logs, and `test_a_normal_route_still_is_logged` guards that exclusion from widening. This is the important one. Today the access log stores a raw IP per request. Without the exclusion, the *one* endpoint whose whole job is turning an IP into a place would leave a stored, timestamped, joinable (IP, approximate-location) pair on disk and in Loki. That artefact — not the transient lookup — is what would actually constitute keeping location data about identifiable people. The exclusion is not defence in depth; it is the substance of the claim. This does **not** depend on the separate access-log IP-truncation work. That change is still worth doing, and it composes: truncation reduces what the *other* routes retain, while this exclusion means the geoip route retains nothing at all either way. - `Cache-Control: no-store, private` + `Vary: *` on the response, so it cannot be stored or reused by any intermediary. - The endpoint takes **no parameters**. You cannot ask it about somebody else's address; the only input is the connection you already own. `?ip=` is ignored (tested). ### Why no consent banner is needed — and why the analytics work is different Under GDPR the lookup is **transient processing to deliver a service the user explicitly asked for**. The visitor pressed "Use my location" *wanting* local weather; when the browser can't supply it, deriving a coarse city from the connection they are already using to reach us is a proportionate substitute for the same, requested purpose. Nothing is stored, nothing is profiled, nothing is shared with a third party (the database is a file we host — no processor relationship, no international transfer, no disclosure to name in the policy). The data subject can override the result with one tap, and the result is explicitly labelled a guess rather than presented as a finding about them. **ePrivacy Art. 5(3) is not engaged at all**, and this is the cleanest part of the argument. That article governs *storing information on, or gaining access to information stored in, a user's terminal equipment* — which is why cookies, localStorage and device fingerprinting need consent regardless of whether the data is personal. An IP address arrives in the request headers as an unavoidable property of routing a packet. We neither store anything on the device nor read anything from it. There is nothing for a consent banner to be about. **Contrast with the analytics session-id work.** That one deliberately *writes an identifier onto the visitor's device* in order to correlate their actions across a session. That is squarely Art. 5(3) storage/access, it is not strictly necessary to deliver a service the user requested, and consent is therefore required — hence `consent.js` and the consent gate on that branch. The two features look adjacent ("both touch visitor data") and are legally opposite: one reads a routing header and forgets it, the other plants a durable identifier. Keep them separate in the code and in the privacy copy, and never let the geoip fallback be folded behind the analytics consent gate — that would imply consent is what makes it lawful, which is the wrong basis and would silently break the fallback for everyone who declines analytics. ### The privacy page follows the flag `frontend/templates/privacy.html.j2` currently says, in the visitor's own words: *"Thermograph never looks up your location from your IP address."* Turning this feature on makes that sentence **false**, and a stale privacy statement is a worse outcome than not shipping the feature. So the paragraph is switched on `THERMOGRAPH_GEOIP`, which both containers read from the same `/etc/thermograph.env` (`content.py`'s `geoip_enabled()`). The statement and the behaviour flip in the same deploy, and the frontend unit tests assert both branches. The frontend reads the flag for *nothing else* — it does not gate any behaviour on it. --- ## 6. Graceful degradation Every one of these returns `None` → HTTP 204 → the client does exactly what the site does today. There is no failure mode worse than the status quo. | Case | Handled where | Behaviour | | --- | --- | --- | | Flag unset | `enabled()` | 204 | | Flag set, no database file | `enabled()` (`os.path.isfile`) | 204 | | `maxminddb` not installed | lazy import in `_open()` | 204 | | Truncated / corrupt / half-written file | `_open()` try/except | 204 (tested) | | Private ranges (10/8, 172.16/12, 192.168/16) | `parse_ip` | 204 — every LAN-dev request | | Loopback, link-local, multicast, reserved, unspecified | `parse_ip` | 204 | | **CGNAT 100.64/10** | `parse_ip` + explicit `_EXTRA_RESERVED` | 204 — common on mobile carriers | | IPv6 (global) | `parse_ip` | looked up normally; MMDB covers v6 | | IPv6 ULA `fc00::/7`, doc `2001:db8::/32` | `_EXTRA_RESERVED` | 204 | | IPv4-mapped IPv6 `::ffff:a.b.c.d` | `parse_ip` unwraps | treated as the v4 address | | `1.2.3.4:5678`, `[v6]:443`, `%zone` suffixes | `parse_ip` | normalised | | Missing `X-Forwarded-For` | `_client_ip` falls back to peer | peer is loopback behind Caddy → 204 | | A whole XFF chain passed as one string | `parse_ip` | rejected (the route already takes the left-most hop) | | No record for the address | `lookup()` | 204 | | Record with a country but **no city** | `suggestion_from_record` | 204 — the VPN/proxy case | | Record whose stated radius > 100 km | `suggestion_from_record` | 204 | | Reader raises on a bad node | `lookup()` try/except | 204 | The explicit `_EXTRA_RESERVED` list is not redundant with `ipaddress.is_global`: the exact membership of `is_private`/`is_global` has moved between CPython releases (100.64/10 in particular). Which Python the container happens to run must not decide whether we try to geolocate a carrier-NAT address. --- ## 7. The UI — presenting a guess as a guess Copy shipped in `frontend/static/geoip.js`: > **Showing weather near Rotterdam, South Holland.** > That is a rough guess from your internet connection, not from your device — it > can be tens of kilometres out. *IP Geolocation by DB-IP* > > **[ Not right? Choose your spot ]** Design rules, and why each one: 1. **Named as a guess, in the same breath as the result.** Not a footnote, not a tooltip. City-level accuracy is ~50 km against a ~2 mile grid cell — the product's own unit of precision is 25× finer than the input. 2. **The correction is a permanent, full-width, ≥44 px control**, not a dismissible toast. If the guess is wrong, fixing it must be the most obvious thing on screen. 3. **A guess is never persisted.** `updateHash()` returns early while `approx` is true: no `localStorage` write, no URL hash. A guess that became sticky would be indistinguishable from a choice on the next visit, and a guessed URL would propagate a rough location the sender never picked when shared. Only an explicit pick is remembered. 4. **The headings are rewritten while approximate.** The hero says "Roughly near Rotterdam" instead of "Today in …", and the results heading says "Near Rotterdam" with an "approximate — from your connection, not your device" tag. This matters more than it sounds: the guessed cell centre gets reverse-geocoded like any other point, which would otherwise name a *neighbourhood* — presenting street-level precision for a city-level input. 5. **Visually recessed** — `--surface-2`, dashed border, muted note text. It reads as an aside, not as an answer. 6. **It never pre-empts the permission prompt.** It runs *after* a declined/failed prompt, or on load only where the browser will not offer geolocation at all (insecure context) and nothing is remembered. 7. **Never for a visitor who already has a place.** Theirs is better. Attribution rides *with the result*, which is what CC BY 4.0 and DB-IP's own wording ask for ("on pages displaying results") — and it is derived from the installed file's metadata, so swapping databases changes the credit automatically. ### Interaction with `home.locate` `home.locate` still means exactly what it means today: "someone tapped *Use my location*", whatever the outcome. It is unchanged, so the existing series stays comparable across this change. Two new allowlisted events sit beside it: - `home.geoip_shown` — a suggestion was rendered. `home.locate` minus `home.geoip_shown` is roughly the population still hitting the dead end. - `home.geoip_fixed` — the visitor corrected the guess. This is the accuracy signal: a high ratio means the database is putting people in the wrong place and the feature is doing harm, which is the number to watch after rollout. Both are aggregate counts through the existing beacon, with **no location dimension** — the suggestion itself is never recorded anywhere. --- ## 8. Database lifecycle **On the host, refreshed by a systemd timer, bind-mounted read-only.** Not baked into the image. - `infra/deploy/geoip-refresh.sh` — resolves the current month's DB-IP release (falling back to last month's, since the new file appears partway through the 1st), downloads, gunzips, **verifies it opens and answers for a known public address**, and only then swaps it in atomically with `install`+`mv`. Exits 0 on "already current". A failure leaves the existing file untouched — a failed refresh must never be worse than a stale one. - `geoip-refresh.timer` — `OnCalendar=Mon 04:17 UTC`, `RandomizedDelaySec=2h`, `Persistent=true`. Weekly rather than monthly even though DB-IP publishes monthly: a no-op run costs one HEAD request, and it means a missed or failed run self-heals within a week. (It also lands inside MaxMind's 30-day deletion window without further thought, should you switch.) - `infra/docker-compose.yml` mounts `${THERMOGRAPH_GEOIP_HOST_DIR:-/var/lib/thermograph/geoip}:/geoip:ro` into the backend. Read-only on purpose — the app must never be able to write it. - `geoip.py` re-opens when the file's `(mtime, size)` changes, so a refresh is picked up **without a restart or a deploy**. - `MODE_MMAP`: the 125 MB file costs page cache, not process RSS, and is shared across the four uvicorn workers on a host. **Why not build-time.** It is a 125 MB data file on its own release cadence, not code. Baking it in adds 125 MB to every image pull, freezes freshness to whenever the last deploy happened, ties a data refresh to a code deploy, and (under GeoLite's EULA) cannot honour the 30-day deletion requirement on its own. **Why not object storage.** We already have the bucket and could pull from it, but that adds credentials and a network dependency to fetch a file that is public and unauthenticated at the source. If a mirror is wanted later — to stop depending on DB-IP's uptime, or to pin exactly which release is live across the fleet — point `GEOIP_URL` at the bucket; the script needs no other change. **Licence key storage.** With DB-IP: **none needed**, which is a large part of the recommendation. If MaxMind is chosen instead, the key goes in the SOPS vault (`infra/deploy/secrets/{prod,beta}.yaml`), renders into `/etc/thermograph.env` like every other secret, and `geoip-refresh.service` already picks it up via its `EnvironmentFile=` — set `GEOIP_URL` to the permalink that embeds it. Note the key would then be in a *URL*, so keep it out of any log line that echoes the URL: `geoip-refresh.sh` logs `$url`, and that line must be changed before using a key-bearing URL. **Flagged as a real footgun if you pick MaxMind.** --- ## 9. Interaction with caching / CDN - **The homepage stays byte-identical for every visitor** — the container is server-rendered empty and hidden. `test_home_html_is_identical_regardless_of_client_address` asserts the body *and* the ETag match across different `X-Forwarded-For` values. That property is what keeps the homepage safe to cache, and it is the main reason the lookup is not in SSR. - **The endpoint is `no-store, private` + `Vary: *`** and answers 204 far more often than 200. Nothing about it is shareable between visitors. - **A CDN in front of the apex stays viable.** If Cloudflare (or a Caddy `cache` directive) is added later, the rule is one line: never cache `/api/*`. That is already the sane default and already how Caddy path-splits. - **The existing frontend caches are untouched.** `api_client.py`'s TTL/LRU cache never sees this request (it is browser→backend, not frontend→backend), and `static/cache.js`'s IndexedDB layer is keyed by `/cell` URLs. A guessed location produces ordinary `/api/v2/grade` traffic for an ordinary cell, so it warms and reuses the same caches as any other place — no new cache keyspace. - **One second-order effect worth naming:** guessed locations concentrate on city centroids, so they will warm the *same* handful of cells repeatedly. That is good for us (high cache hit rate, no extra Open-Meteo quota) but it does mean per-cell request counts stop being a clean proxy for distinct users near that cell. --- ## 10. Rollout 1. Merge with the flag off. Nothing changes — that is the point, and the tests assert it. 2. On **beta only**: install the timer, run `geoip-refresh.sh` once by hand, confirm `/var/lib/thermograph/geoip/city.mmdb` exists and `curl -s -o /dev/null -w '%{http_code}' https://beta.thermograph.org/api/v2/geoip` still returns 204 (flag still off). 3. Set `THERMOGRAPH_GEOIP=1` in beta's vault entry, redeploy **both** containers (backend for the lookup, frontend for the privacy copy). 4. Verify by hand: decline the permission prompt; confirm the banner appears, the correction button works, the guess is *not* in `localStorage` and *not* in the URL; confirm `/privacy` now describes the fallback; confirm no `"cat":"geoip"` line and no matching IP appears in the access log. 5. Watch `home.geoip_fixed / home.geoip_shown` for a week. If a large share of visitors are correcting the guess, the database is putting people in the wrong place — turn it off rather than tuning copy. 6. Only then, prod. Rollback is `THERMOGRAPH_GEOIP=0` and a redeploy; or just delete the database file, which disables it without touching config. --- ## 11. Decisions needed from the operator 1. **Which database?** Recommendation: **DB-IP IP to City Lite** (no account, no key, no retention clock, CC BY 4.0). The alternative is GeoLite2 City for `accuracy_radius` and weekly updates, at the cost of a vault-managed licence key and a 30-day deletion obligation. Code supports either unchanged. 2. **Attribution placement.** Currently in the suggestion banner itself, beside the result. Alternatives: also on `/about`, or only on `/about`. DB-IP's stated wording asks for it "on pages displaying results", which the banner satisfies — confirm you're happy with a small credit line inside the hero. 3. **SSR or client-side?** Recommendation: **client-side** (§4). Overruling this means accepting a location-varying homepage; if you do, put it on a separate non-canonical route, never on cached `/`. 4. **Refresh cadence.** Currently weekly-with-no-op (§8). Alternatives: monthly to match DB-IP's release cadence exactly, or daily. Weekly is the self-healing middle. 5. **Auto-suggest in insecure contexts?** Today the prototype suggests on load when the browser won't offer geolocation at all (LAN dev over plain HTTP) and nothing is remembered, since there is no prompt to decline. That is the only path where a lookup happens without an explicit user action. Prod is HTTPS so it never triggers there — but say if you'd rather it *never* happen without a tap, and it becomes a one-line change. 6. **Threshold for calling it off.** Pick the `home.geoip_fixed / home.geoip_shown` ratio at which the feature gets turned off, *before* rollout rather than after — otherwise the number will always look explainable. --- ## 12. Things I think are bad ideas — flagged, not implemented - **Don't put the lookup in the SSR homepage.** §4. A location-varying canonical page is a cache-poisoning and SEO hazard, and it geolocates the majority of visitors who already have a saved place and will never use it. - **Don't persist the guess.** Not to `localStorage`, not to the URL, not to an account preference. It is not a choice and must never become one silently. The branch enforces this in `updateHash()`. - **Don't use a third-party geolocation API**, even a "privacy-friendly" one. Sending visitor IPs to another company creates a processor relationship and a disclosure obligation, for a file we can just host. This was already the brief's requirement; restating it because it is the single most tempting shortcut here. - **Don't extend this to country-level personalisation** (units, language, default city) on the same lookup. Unit selection already works off the picked location's country code, and quietly widening an IP lookup from "a fallback the user asked for" to "we adapt the site to you" changes the legal analysis in §5 — it stops being strictly necessary for a requested service. - **Don't reuse the derived city anywhere it could be stored.** No metrics dimension, no audit field, no "popular cities" table fed from guesses. The moment a derived location is written next to anything timestamped, the privacy claim in §5 stops being true. - **Don't fold this behind the analytics consent gate.** §5. Different legal basis; gating it on consent both implies the wrong basis and breaks the fallback for everyone who declines analytics. - **Don't log `$url` once a MaxMind key-bearing permalink is in use** (§8). - **Be honest that this is a downgrade, not a feature.** A city-level guess is 25× coarser than the grid the whole product is built on. It exists so a visitor who declined isn't stranded — not because IP location is good. If the correction rate says people don't want it, delete it rather than defend it.