thermograph/GEOIP-APPROX-LOCATION.md

Approximate-location fallback — design + prototype

Status: prototype on feat/geoip-approx-location, feature-flagged OFF, not deployed and not merged. Nothing in this branch changes production behaviour until THERMOGRAPH_GEOIP is set and a database file exists on the host.

This doc is the decision record. If it survives review it should move to thermograph-docs (per the root CLAUDE.md: cross-cutting decision docs live there); it sits here for now so the reasoning ships with the code it describes.

---

1. The problem

Location today comes from exactly one source: navigator.geolocation.getCurrentPosition in frontend/static/app.js, gated on window.isSecureContext. If the visitor declines the permission prompt, the prompt times out, or the browser won't offer geolocation at all, the site has no location and the copy says "Pick a spot on the map instead."

That is a dead end for the exact visitor the homepage is written for — someone who arrived to answer "how unusual is my weather?" and has just been handed a world map. This change gives that person something local, honestly labelled.

2. What was built

Piece	File
Lookup module (flag, parsing, rejection rules, attribution)	backend/data/geoip.py
Endpoint GET /api/v2/geoip	backend/web/app.py (api_geoip)
Access-log exclusion + traffic category	backend/web/app.py, backend/core/metrics.py
Event names home.geoip_shown / home.geoip_fixed	backend/core/metrics.py
Client fetch + suggestion banner	frontend/static/geoip.js
Wiring, non-persistence, honest headings	frontend/static/app.js
Banner container (empty, hidden)	frontend/templates/home.html.j2
Banner styles	frontend/static/style.css
Flag-aware privacy copy	frontend/content.py, frontend/templates/privacy.html.j2
Host refresh job + timer	infra/deploy/geoip-refresh.{sh,service,timer}
Read-only DB mount + env plumbing	infra/docker-compose.yml, infra/deploy/thermograph.env.example
Tests	backend/tests/data/test_geoip.py, backend/tests/web/test_geoip_api.py, frontend/tests/unit/test_geoip_ui.py

cd backend && make test → 449 passed, 8 skipped (was 386/7). cd frontend && make test-unit → 41 passed.

---

3. Database choice — recommendation: DB-IP IP to City Lite

	DB-IP IP to City Lite	MaxMind GeoLite2 City	IP2Location LITE DB5/DB11
Licence	CC BY 4.0	GeoLite End User License Agreement (incorporates CC BY-SA 4.0 terms)	CC BY-SA 4.0 / "free with attribution"
Account needed	No — direct download URL	Yes — account + licence key	Yes — free account
Secret to manage	None	Licence key (→ SOPS vault, → CI, → rotation)	Account credentials
Attribution	<a href="https://db-ip.com">IP Geolocation by DB-IP</a> on pages showing results	"This product includes GeoLite Data created by MaxMind, available from https://www.maxmind.com"	"uses the IP2Location LITE database for IP geolocation", linked
Retention obligation	none	must delete a database within 30 days of a newer release; 30 downloads/day cap	none
Update cadence	monthly	weekly (Tue/Fri)	monthly
Size (current release)	125 MB MMDB (685 MB CSV), ~8.05 M records	~60–70 MB MMDB	69 MB BIN (IPv4) + 171 MB BIN (IPv6), separate files
Format / reader	MMDB → maxminddb (pure-Python, mmap)	MMDB → maxminddb / geoip2	proprietary BIN → vendor IP2Location package (or CSV you index yourself)
Accuracy fields	city + lat/lon, no accuracy_radius	city + lat/lon + accuracy_radius	city + lat/lon

Recommend DB-IP IP to City Lite. Reasoning, in priority order:

1. No credential. MaxMind means a licence key in the SOPS vault, in the deploy path, and in a rotation story — new secret surface for a feature whose entire justification is privacy hygiene. DB-IP needs none, so geoip-refresh.sh reads nothing from the vault at all.
2. No retention obligation. GeoLite's EULA requires deleting a database within 30 days of a newer release. That is a compliance clock on an ops job, and a missed job becomes a licence breach rather than merely stale data. CC BY 4.0 has no such clause.
3. Same format, same reader. Both are MMDB, so this is a swap of one file and one URL, not a code change — geoip.py reads the raw record dict and derives its attribution line from the file's own metadata. If accuracy turns out to be materially worse in practice, moving to GeoLite2 is a config change plus a vault entry, and the code is already written to accept it.
4. Attribution is a link, not a legal paragraph. One muted anchor in the suggestion banner satisfies CC BY 4.0 and DB-IP's stated wording.

Against IP2Location LITE: a proprietary BIN format needing the vendor's own Python package, IPv4 and IPv6 as separate databases (two files, two readers, two refresh paths), an account to download, and CC BY-SA — a share-alike term that is noise to reason about for a file we merely read.

Cost of the DB-IP choice, stated plainly: the Lite file carries no accuracy_radius, so the "reject anything the database admits is too coarse" guard degrades to "reject anything with no city name". That is still the guard that matters (a country centroid is the damaging case), but it is weaker. suggestion_from_record() applies the radius rule whenever the field is present, so switching to GeoLite2 turns the stronger guard back on for free.

Reader library: maxminddb (pinned 2.8.2 in backend/requirements.txt), not geoip2. geoip2 adds a model layer over the same reader and type-checks database_type against MaxMind's own names; we read four fields out of a plain dict, and staying on maxminddb keeps DB-IP and MaxMind files equally usable. It is imported lazily inside geoip.py, so a missing wheel disables the feature rather than breaking boot.

---

4. Where it hooks in — recommendation: client-side, on demand

Two options were on the table.

A. Server-side during SSR of /. frontend/content.py's home_page() would look up the IP and render the guess into the first paint.

B. Client-side after geolocation is declined/unavailable. The browser calls GET /api/v2/geoip and swaps the banner in. This is what the prototype does.

Why B

Caching is the deciding factor, and the current setup is more fragile than it looks. frontend/content.py's _respond_html() returns HTML with a weak ETag (W/"<sha1 of the rendered body>") and no Cache-Control and no Vary. The host Caddy (infra/deploy/Caddyfile) only does encode + reverse_proxy — no cache module today. So right now nothing caches the homepage, but nothing declares that it mustn't either.

Fold a location into that response and three things break at once:

• The ETag becomes a function of the visitor's city. Two visitors flap each other's validators; conditional requests stop being useful, and a returning visitor whose IP maps elsewhere gets a spurious 200 instead of a 304.
• Any cache added later — a Caddy cache directive, Cloudflare in front of the apex, a corporate proxy on the visitor's side — will serve one visitor's city to another, because the response carries nothing saying it varies. There is no correct Vary header for this: it varies on X-Forwarded-For, which is not a header a cache keys on, and Vary: * would make the homepage permanently uncacheable for everyone to serve a fallback that only a minority need.
• The SSR path would also have to carry the client IP over the extra frontend→backend hop (api_client.py currently forwards only Host and X-Forwarded-Proto). That means propagating the IP into a second process's request path — more places it can end up in a log, for no benefit.

The first-paint argument is weaker here than it looks, because the homepage hero is already server-rendered with real content: the most unusual city we're currently tracking, with its grade, band colour and percentile. There is no blank space for the guess to fill. The suggestion swaps into a page that is already complete, and it costs one small JSON request with no data dependency in front of it.

And SSR would do the lookup for the wrong people. The server cannot see localStorage, so it cannot know that most returning visitors already have a saved place that is strictly better than any guess. An SSR implementation would geolocate every homepage request, including the majority who will never use the result — the opposite of "touch the IP as little as possible". Option B only runs after the visitor has demonstrably ended up with nothing.

What B costs

• Needs JavaScript. A no-JS visitor sees today's behaviour. Acceptable: the entire interactive tool already needs JS.
• One extra round trip after the decline, ~200 bytes each way.
• A 204 for everyone in the fallback path when the feature is off. The alternative — giving the frontend its own copy of the flag to decide whether to fetch — is two flags to keep in sync and a new way for the UI and the server to disagree. One flag, one 204, is the better trade.

If you overrule this and want SSR anyway

Do it as a separate route (/near or ?near=1), not on the cached /, and give that route Cache-Control: private, no-store. Never make the canonical, indexable homepage location-varying — that is also an SEO hazard: a crawler in one datacentre would index a page claiming a city, and different crawler IPs would see different "canonical" content.

---

5. Privacy design

The IP is never persisted — mechanically, not by convention

• geoip.lookup(ip) takes a string, memory-maps a file, returns a dict. It writes nothing, logs nothing, and never raises.

• api_geoip uses no RunAudit (which would record the derived lat/lon) and adds no metric dimension (the counters are aggregate by category).

• The route is excluded from the access log. metrics.classify_inbound() now returns a dedicated "geoip" category and the middleware in web/app.py skips audit.log_access for it. Every other route still logs, and test_a_normal_route_still_is_logged guards that exclusion from widening.

  This is the important one. Today the access log stores a raw IP per request. Without the exclusion, the one endpoint whose whole job is turning an IP into a place would leave a stored, timestamped, joinable (IP, approximate-location) pair on disk and in Loki. That artefact — not the transient lookup — is what would actually constitute keeping location data about identifiable people. The exclusion is not defence in depth; it is the substance of the claim.

  This does not depend on the separate access-log IP-truncation work. That change is still worth doing, and it composes: truncation reduces what the other routes retain, while this exclusion means the geoip route retains nothing at all either way.

• Cache-Control: no-store, private + Vary: * on the response, so it cannot be stored or reused by any intermediary.

• The endpoint takes no parameters. You cannot ask it about somebody else's address; the only input is the connection you already own. ?ip= is ignored (tested).

Why no consent banner is needed — and why the analytics work is different

Under GDPR the lookup is transient processing to deliver a service the user explicitly asked for. The visitor pressed "Use my location" wanting local weather; when the browser can't supply it, deriving a coarse city from the connection they are already using to reach us is a proportionate substitute for the same, requested purpose. Nothing is stored, nothing is profiled, nothing is shared with a third party (the database is a file we host — no processor relationship, no international transfer, no disclosure to name in the policy). The data subject can override the result with one tap, and the result is explicitly labelled a guess rather than presented as a finding about them.

ePrivacy Art. 5(3) is not engaged at all, and this is the cleanest part of the argument. That article governs storing information on, or gaining access to information stored in, a user's terminal equipment — which is why cookies, localStorage and device fingerprinting need consent regardless of whether the data is personal. An IP address arrives in the request headers as an unavoidable property of routing a packet. We neither store anything on the device nor read anything from it. There is nothing for a consent banner to be about.

Contrast with the analytics session-id work. That one deliberately writes an identifier onto the visitor's device in order to correlate their actions across a session. That is squarely Art. 5(3) storage/access, it is not strictly necessary to deliver a service the user requested, and consent is therefore required — hence consent.js and the consent gate on that branch. The two features look adjacent ("both touch visitor data") and are legally opposite: one reads a routing header and forgets it, the other plants a durable identifier. Keep them separate in the code and in the privacy copy, and never let the geoip fallback be folded behind the analytics consent gate — that would imply consent is what makes it lawful, which is the wrong basis and would silently break the fallback for everyone who declines analytics.

The privacy page follows the flag

frontend/templates/privacy.html.j2 currently says, in the visitor's own words: "Thermograph never looks up your location from your IP address." Turning this feature on makes that sentence false, and a stale privacy statement is a worse outcome than not shipping the feature.

So the paragraph is switched on THERMOGRAPH_GEOIP, which both containers read from the same /etc/thermograph.env (content.py's geoip_enabled()). The statement and the behaviour flip in the same deploy, and the frontend unit tests assert both branches. The frontend reads the flag for nothing else — it does not gate any behaviour on it.

---

6. Graceful degradation

Every one of these returns None → HTTP 204 → the client does exactly what the site does today. There is no failure mode worse than the status quo.

Case	Handled where	Behaviour
Flag unset	enabled()	204
Flag set, no database file	enabled() (os.path.isfile)	204
maxminddb not installed	lazy import in _open()	204
Truncated / corrupt / half-written file	_open() try/except	204 (tested)
Private ranges (10/8, 172.16/12, 192.168/16)	parse_ip	204 — every LAN-dev request
Loopback, link-local, multicast, reserved, unspecified	parse_ip	204
CGNAT 100.64/10	parse_ip + explicit _EXTRA_RESERVED	204 — common on mobile carriers
IPv6 (global)	parse_ip	looked up normally; MMDB covers v6
IPv6 ULA fc00::/7, doc 2001:db8::/32	_EXTRA_RESERVED	204
IPv4-mapped IPv6 ::ffff:a.b.c.d	parse_ip unwraps	treated as the v4 address
1.2.3.4:5678, [v6]:443, %zone suffixes	parse_ip	normalised
Missing X-Forwarded-For	_client_ip falls back to peer	peer is loopback behind Caddy → 204
A whole XFF chain passed as one string	parse_ip	rejected (the route already takes the left-most hop)
No record for the address	lookup()	204
Record with a country but no city	suggestion_from_record	204 — the VPN/proxy case
Record whose stated radius > 100 km	suggestion_from_record	204
Reader raises on a bad node	lookup() try/except	204

The explicit _EXTRA_RESERVED list is not redundant with ipaddress.is_global: the exact membership of is_private/is_global has moved between CPython releases (100.64/10 in particular). Which Python the container happens to run must not decide whether we try to geolocate a carrier-NAT address.

---

7. The UI — presenting a guess as a guess

Copy shipped in frontend/static/geoip.js:

Showing weather near Rotterdam, South Holland. That is a rough guess from your internet connection, not from your device — it can be tens of kilometres out. IP Geolocation by DB-IP

[ Not right? Choose your spot ]

Design rules, and why each one:

1. Named as a guess, in the same breath as the result. Not a footnote, not a tooltip. City-level accuracy is ~50 km against a ~2 mile grid cell — the product's own unit of precision is 25× finer than the input.
2. The correction is a permanent, full-width, ≥44 px control, not a dismissible toast. If the guess is wrong, fixing it must be the most obvious thing on screen.
3. A guess is never persisted. updateHash() returns early while approx is true: no localStorage write, no URL hash. A guess that became sticky would be indistinguishable from a choice on the next visit, and a guessed URL would propagate a rough location the sender never picked when shared. Only an explicit pick is remembered.
4. The headings are rewritten while approximate. The hero says "Roughly near Rotterdam" instead of "Today in …", and the results heading says "Near Rotterdam" with an "approximate — from your connection, not your device" tag. This matters more than it sounds: the guessed cell centre gets reverse-geocoded like any other point, which would otherwise name a neighbourhood — presenting street-level precision for a city-level input.
5. Visually recessed — --surface-2, dashed border, muted note text. It reads as an aside, not as an answer.
6. It never pre-empts the permission prompt. It runs after a declined/failed prompt, or on load only where the browser will not offer geolocation at all (insecure context) and nothing is remembered.
7. Never for a visitor who already has a place. Theirs is better.

Attribution rides with the result, which is what CC BY 4.0 and DB-IP's own wording ask for ("on pages displaying results") — and it is derived from the installed file's metadata, so swapping databases changes the credit automatically.

Interaction with home.locate

home.locate still means exactly what it means today: "someone tapped Use my location", whatever the outcome. It is unchanged, so the existing series stays comparable across this change. Two new allowlisted events sit beside it:

• home.geoip_shown — a suggestion was rendered. home.locate minus home.geoip_shown is roughly the population still hitting the dead end.
• home.geoip_fixed — the visitor corrected the guess. This is the accuracy signal: a high ratio means the database is putting people in the wrong place and the feature is doing harm, which is the number to watch after rollout.

Both are aggregate counts through the existing beacon, with no location dimension — the suggestion itself is never recorded anywhere.

---

8. Database lifecycle

On the host, refreshed by a systemd timer, bind-mounted read-only. Not baked into the image.

• infra/deploy/geoip-refresh.sh — resolves the current month's DB-IP release (falling back to last month's, since the new file appears partway through the 1st), downloads, gunzips, verifies it opens and answers for a known public address, and only then swaps it in atomically with install+mv. Exits 0 on "already current". A failure leaves the existing file untouched — a failed refresh must never be worse than a stale one.
• geoip-refresh.timer — OnCalendar=Mon 04:17 UTC, RandomizedDelaySec=2h, Persistent=true. Weekly rather than monthly even though DB-IP publishes monthly: a no-op run costs one HEAD request, and it means a missed or failed run self-heals within a week. (It also lands inside MaxMind's 30-day deletion window without further thought, should you switch.)
• infra/docker-compose.yml mounts ${THERMOGRAPH_GEOIP_HOST_DIR:-/var/lib/thermograph/geoip}:/geoip:ro into the backend. Read-only on purpose — the app must never be able to write it.
• geoip.py re-opens when the file's (mtime, size) changes, so a refresh is picked up without a restart or a deploy.
• MODE_MMAP: the 125 MB file costs page cache, not process RSS, and is shared across the four uvicorn workers on a host.

Why not build-time. It is a 125 MB data file on its own release cadence, not code. Baking it in adds 125 MB to every image pull, freezes freshness to whenever the last deploy happened, ties a data refresh to a code deploy, and (under GeoLite's EULA) cannot honour the 30-day deletion requirement on its own.

Why not object storage. We already have the bucket and could pull from it, but that adds credentials and a network dependency to fetch a file that is public and unauthenticated at the source. If a mirror is wanted later — to stop depending on DB-IP's uptime, or to pin exactly which release is live across the fleet — point GEOIP_URL at the bucket; the script needs no other change.

Licence key storage. With DB-IP: none needed, which is a large part of the recommendation. If MaxMind is chosen instead, the key goes in the SOPS vault (infra/deploy/secrets/{prod,beta}.yaml), renders into /etc/thermograph.env like every other secret, and geoip-refresh.service already picks it up via its EnvironmentFile= — set GEOIP_URL to the permalink that embeds it. Note the key would then be in a URL, so keep it out of any log line that echoes the URL: geoip-refresh.sh logs $url, and that line must be changed before using a key-bearing URL. Flagged as a real footgun if you pick MaxMind.

---

9. Interaction with caching / CDN

• The homepage stays byte-identical for every visitor — the container is server-rendered empty and hidden. test_home_html_is_identical_regardless_of_client_address asserts the body and the ETag match across different X-Forwarded-For values. That property is what keeps the homepage safe to cache, and it is the main reason the lookup is not in SSR.
• The endpoint is no-store, private + Vary: * and answers 204 far more often than 200. Nothing about it is shareable between visitors.
• A CDN in front of the apex stays viable. If Cloudflare (or a Caddy cache directive) is added later, the rule is one line: never cache /api/*. That is already the sane default and already how Caddy path-splits.
• The existing frontend caches are untouched. api_client.py's TTL/LRU cache never sees this request (it is browser→backend, not frontend→backend), and static/cache.js's IndexedDB layer is keyed by /cell URLs. A guessed location produces ordinary /api/v2/grade traffic for an ordinary cell, so it warms and reuses the same caches as any other place — no new cache keyspace.
• One second-order effect worth naming: guessed locations concentrate on city centroids, so they will warm the same handful of cells repeatedly. That is good for us (high cache hit rate, no extra Open-Meteo quota) but it does mean per-cell request counts stop being a clean proxy for distinct users near that cell.

---

10. Rollout

1. Merge with the flag off. Nothing changes — that is the point, and the tests assert it.
2. On beta only: install the timer, run geoip-refresh.sh once by hand, confirm /var/lib/thermograph/geoip/city.mmdb exists and curl -s -o /dev/null -w '%{http_code}' https://beta.thermograph.org/api/v2/geoip still returns 204 (flag still off).
3. Set THERMOGRAPH_GEOIP=1 in beta's vault entry, redeploy both containers (backend for the lookup, frontend for the privacy copy).
4. Verify by hand: decline the permission prompt; confirm the banner appears, the correction button works, the guess is not in localStorage and not in the URL; confirm /privacy now describes the fallback; confirm no "cat":"geoip" line and no matching IP appears in the access log.
5. Watch home.geoip_fixed / home.geoip_shown for a week. If a large share of visitors are correcting the guess, the database is putting people in the wrong place — turn it off rather than tuning copy.
6. Only then, prod.

Rollback is THERMOGRAPH_GEOIP=0 and a redeploy; or just delete the database file, which disables it without touching config.

---

11. Decisions needed from the operator

1. Which database? Recommendation: DB-IP IP to City Lite (no account, no key, no retention clock, CC BY 4.0). The alternative is GeoLite2 City for accuracy_radius and weekly updates, at the cost of a vault-managed licence key and a 30-day deletion obligation. Code supports either unchanged.
2. Attribution placement. Currently in the suggestion banner itself, beside the result. Alternatives: also on /about, or only on /about. DB-IP's stated wording asks for it "on pages displaying results", which the banner satisfies — confirm you're happy with a small credit line inside the hero.
3. SSR or client-side? Recommendation: client-side (§4). Overruling this means accepting a location-varying homepage; if you do, put it on a separate non-canonical route, never on cached /.
4. Refresh cadence. Currently weekly-with-no-op (§8). Alternatives: monthly to match DB-IP's release cadence exactly, or daily. Weekly is the self-healing middle.
5. Auto-suggest in insecure contexts? Today the prototype suggests on load when the browser won't offer geolocation at all (LAN dev over plain HTTP) and nothing is remembered, since there is no prompt to decline. That is the only path where a lookup happens without an explicit user action. Prod is HTTPS so it never triggers there — but say if you'd rather it never happen without a tap, and it becomes a one-line change.
6. Threshold for calling it off. Pick the home.geoip_fixed / home.geoip_shown ratio at which the feature gets turned off, before rollout rather than after — otherwise the number will always look explainable.

---

12. Things I think are bad ideas — flagged, not implemented

• Don't put the lookup in the SSR homepage. §4. A location-varying canonical page is a cache-poisoning and SEO hazard, and it geolocates the majority of visitors who already have a saved place and will never use it.
• Don't persist the guess. Not to localStorage, not to the URL, not to an account preference. It is not a choice and must never become one silently. The branch enforces this in updateHash().
• Don't use a third-party geolocation API, even a "privacy-friendly" one. Sending visitor IPs to another company creates a processor relationship and a disclosure obligation, for a file we can just host. This was already the brief's requirement; restating it because it is the single most tempting shortcut here.
• Don't extend this to country-level personalisation (units, language, default city) on the same lookup. Unit selection already works off the picked location's country code, and quietly widening an IP lookup from "a fallback the user asked for" to "we adapt the site to you" changes the legal analysis in §5 — it stops being strictly necessary for a requested service.
• Don't reuse the derived city anywhere it could be stored. No metrics dimension, no audit field, no "popular cities" table fed from guesses. The moment a derived location is written next to anything timestamped, the privacy claim in §5 stops being true.
• Don't fold this behind the analytics consent gate. §5. Different legal basis; gating it on consent both implies the wrong basis and breaks the fallback for everyone who declines analytics.
• Don't log $url once a MaxMind key-bearing permalink is in use (§8).
• Be honest that this is a downgrade, not a feature. A city-level guess is 25× coarser than the grid the whole product is built on. It exists so a visitor who declined isn't stranded — not because IP location is good. If the correction rate says people don't want it, delete it rather than defend it.