Approximate-location fallback from the client IP (feature-flagged off)
All checks were successful
PR build (required check) / changes (pull_request) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Successful in 1m8s
PR build (required check) / build-backend (pull_request) Successful in 1m36s
PR build (required check) / gate (pull_request) Successful in 2s

A visitor who declines browser geolocation currently has no location at all —
the copy sends them to the map picker. This adds an opt-in fallback that
suggests a coarse city from the request's own IP, presented as a guess with a
one-tap correction, so the dead end has a way out.

backend/data/geoip.py does the lookup against a local MMDB file (DB-IP IP to
City Lite or GeoLite2 City — same format, either works, attribution derived
from the file's metadata). Off unless THERMOGRAPH_GEOIP is truthy AND the
database exists AND maxminddb imports; every degraded case — private/reserved/
CGNAT/IPv6-ULA addresses, unparseable input, no record, a country-centroid
record with no city, a record wider than the accuracy limit, a corrupt file —
returns None, which the route answers as 204 and the client treats exactly as
today.

The IP is read in memory and dropped. GET /api/v2/geoip runs no RunAudit,
records no metric dimension, and is excluded from the access log (its own
"geoip" traffic category) so no stored, joinable (IP, location) pair is ever
written. The response is no-store/Vary:* and takes no parameters.

Client-side rather than server-rendered on purpose: the homepage's HTML and
weak ETag must stay byte-identical for every visitor, or any cache added in
front of it can serve one visitor's city to another. The suggestion is fetched
only after a declined/unavailable prompt, and only when nothing is remembered.

A guess is never persisted — no localStorage write, no URL hash — and the hero
and results headings say "roughly near"/"near … approximate" rather than
letting the reverse-geocoded cell name a neighbourhood the lookup never knew.

The /privacy page's "never looks up your location from your IP address"
paragraph now follows the same flag out of the same env file, so the published
statement and the behaviour flip together.

Database lifecycle is a host-side systemd timer (infra/deploy/geoip-refresh.*)
that verifies a download opens and answers before swapping it in atomically,
bind-mounted read-only; geoip.py re-opens on mtime change, so a refresh needs
no restart. GEOIP-APPROX-LOCATION.md carries the database comparison, the
SSR-vs-client argument, the privacy analysis, and the open decisions.
This commit is contained in:
Emi Griffith 2026-07-23 16:22:37 -07:00
parent 16bddb6625
commit 083d3bd0f8
19 changed files with 1694 additions and 4 deletions

489
GEOIP-APPROX-LOCATION.md Normal file
View file

@ -0,0 +1,489 @@
# Approximate-location fallback — design + prototype
Status: **prototype on `feat/geoip-approx-location`, feature-flagged OFF, not
deployed and not merged.** Nothing in this branch changes production behaviour
until `THERMOGRAPH_GEOIP` is set *and* a database file exists on the host.
This doc is the decision record. If it survives review it should move to
`thermograph-docs` (per the root `CLAUDE.md`: cross-cutting decision docs live
there); it sits here for now so the reasoning ships with the code it describes.
---
## 1. The problem
Location today comes from exactly one source: `navigator.geolocation.getCurrentPosition`
in `frontend/static/app.js`, gated on `window.isSecureContext`. If the visitor
declines the permission prompt, the prompt times out, or the browser won't offer
geolocation at all, the site has **no location** and the copy says "Pick a spot
on the map instead."
That is a dead end for the exact visitor the homepage is written for — someone
who arrived to answer "how unusual is *my* weather?" and has just been handed a
world map. This change gives that person something local, honestly labelled.
## 2. What was built
| Piece | File |
| --- | --- |
| Lookup module (flag, parsing, rejection rules, attribution) | `backend/data/geoip.py` |
| Endpoint `GET /api/v2/geoip` | `backend/web/app.py` (`api_geoip`) |
| Access-log exclusion + traffic category | `backend/web/app.py`, `backend/core/metrics.py` |
| Event names `home.geoip_shown` / `home.geoip_fixed` | `backend/core/metrics.py` |
| Client fetch + suggestion banner | `frontend/static/geoip.js` |
| Wiring, non-persistence, honest headings | `frontend/static/app.js` |
| Banner container (empty, hidden) | `frontend/templates/home.html.j2` |
| Banner styles | `frontend/static/style.css` |
| Flag-aware privacy copy | `frontend/content.py`, `frontend/templates/privacy.html.j2` |
| Host refresh job + timer | `infra/deploy/geoip-refresh.{sh,service,timer}` |
| Read-only DB mount + env plumbing | `infra/docker-compose.yml`, `infra/deploy/thermograph.env.example` |
| Tests | `backend/tests/data/test_geoip.py`, `backend/tests/web/test_geoip_api.py`, `frontend/tests/unit/test_geoip_ui.py` |
`cd backend && make test`**449 passed, 8 skipped** (was 386/7).
`cd frontend && make test-unit`**41 passed**.
---
## 3. Database choice — recommendation: **DB-IP IP to City Lite**
| | **DB-IP IP to City Lite** | MaxMind GeoLite2 City | IP2Location LITE DB5/DB11 |
| --- | --- | --- | --- |
| Licence | **CC BY 4.0** | GeoLite End User License Agreement (incorporates CC BY-SA 4.0 terms) | CC BY-SA 4.0 / "free with attribution" |
| Account needed | **No** — direct download URL | **Yes** — account + licence key | **Yes** — free account |
| Secret to manage | **None** | Licence key (→ SOPS vault, → CI, → rotation) | Account credentials |
| Attribution | `<a href="https://db-ip.com">IP Geolocation by DB-IP</a>` on pages showing results | "This product includes GeoLite Data created by MaxMind, available from https://www.maxmind.com" | "uses the IP2Location LITE database for IP geolocation", linked |
| Retention obligation | none | **must delete a database within 30 days of a newer release**; 30 downloads/day cap | none |
| Update cadence | monthly | weekly (Tue/Fri) | monthly |
| Size (current release) | 125 MB MMDB (685 MB CSV), ~8.05 M records | ~6070 MB MMDB | 69 MB BIN (IPv4) + 171 MB BIN (IPv6), separate files |
| Format / reader | **MMDB → `maxminddb`** (pure-Python, mmap) | MMDB → `maxminddb` / `geoip2` | proprietary BIN → vendor `IP2Location` package (or CSV you index yourself) |
| Accuracy fields | city + lat/lon, **no `accuracy_radius`** | city + lat/lon + `accuracy_radius` | city + lat/lon |
**Recommend DB-IP IP to City Lite.** Reasoning, in priority order:
1. **No credential.** MaxMind means a licence key in the SOPS vault, in the
deploy path, and in a rotation story — new secret surface for a feature whose
entire justification is privacy hygiene. DB-IP needs none, so
`geoip-refresh.sh` reads nothing from the vault at all.
2. **No retention obligation.** GeoLite's EULA requires deleting a database
within 30 days of a newer release. That is a *compliance* clock on an ops
job, and a missed job becomes a licence breach rather than merely stale data.
CC BY 4.0 has no such clause.
3. **Same format, same reader.** Both are MMDB, so this is a swap of one file
and one URL, not a code change — `geoip.py` reads the raw record dict and
derives its attribution line from the file's own metadata. If accuracy turns
out to be materially worse in practice, moving to GeoLite2 is a config change
plus a vault entry, and the code is already written to accept it.
4. **Attribution is a link, not a legal paragraph.** One muted anchor in the
suggestion banner satisfies CC BY 4.0 and DB-IP's stated wording.
**Against IP2Location LITE**: a proprietary BIN format needing the vendor's own
Python package, IPv4 and IPv6 as *separate* databases (two files, two readers,
two refresh paths), an account to download, and CC BY-**SA** — a share-alike
term that is noise to reason about for a file we merely read.
**Cost of the DB-IP choice, stated plainly:** the Lite file carries no
`accuracy_radius`, so the "reject anything the database admits is too coarse"
guard degrades to "reject anything with no city name". That is still the guard
that matters (a country centroid is the damaging case), but it is weaker.
`suggestion_from_record()` applies the radius rule whenever the field *is*
present, so switching to GeoLite2 turns the stronger guard back on for free.
**Reader library:** `maxminddb` (pinned `2.8.2` in `backend/requirements.txt`),
not `geoip2`. `geoip2` adds a model layer over the same reader and type-checks
`database_type` against MaxMind's own names; we read four fields out of a plain
dict, and staying on `maxminddb` keeps DB-IP and MaxMind files equally usable.
It is imported *lazily inside* `geoip.py`, so a missing wheel disables the
feature rather than breaking boot.
---
## 4. Where it hooks in — recommendation: **client-side, on demand**
Two options were on the table.
**A. Server-side during SSR of `/`.** `frontend/content.py`'s `home_page()`
would look up the IP and render the guess into the first paint.
**B. Client-side after geolocation is declined/unavailable.** The browser calls
`GET /api/v2/geoip` and swaps the banner in. **This is what the prototype does.**
### Why B
**Caching is the deciding factor, and the current setup is more fragile than it
looks.** `frontend/content.py`'s `_respond_html()` returns HTML with a weak
ETag (`W/"<sha1 of the rendered body>"`) and **no `Cache-Control` and no
`Vary`**. The host Caddy (`infra/deploy/Caddyfile`) only does `encode` +
`reverse_proxy` — no cache module today. So right now nothing caches the
homepage, but nothing *declares* that it mustn't either.
Fold a location into that response and three things break at once:
- The ETag becomes a function of the visitor's city. Two visitors flap each
other's validators; conditional requests stop being useful, and a returning
visitor whose IP maps elsewhere gets a spurious 200 instead of a 304.
- Any cache added later — a Caddy `cache` directive, Cloudflare in front of the
apex, a corporate proxy on the visitor's side — will serve one visitor's city
to another, because the response carries nothing saying it varies. There is no
correct `Vary` header for this: it varies on `X-Forwarded-For`, which is not a
header a cache keys on, and `Vary: *` would make the homepage permanently
uncacheable for everyone to serve a fallback that only a minority need.
- The SSR path would also have to carry the client IP over the extra
frontend→backend hop (`api_client.py` currently forwards only `Host` and
`X-Forwarded-Proto`). That means propagating the IP into a second process's
request path — more places it can end up in a log, for no benefit.
**The first-paint argument is weaker here than it looks**, because the homepage
hero is *already* server-rendered with real content: the most unusual city
we're currently tracking, with its grade, band colour and percentile. There is
no blank space for the guess to fill. The suggestion swaps into a page that is
already complete, and it costs one small JSON request with no data dependency in
front of it.
**And SSR would do the lookup for the wrong people.** The server cannot see
`localStorage`, so it cannot know that most returning visitors already have a
saved place that is strictly better than any guess. An SSR implementation would
geolocate every homepage request, including the majority who will never use the
result — the opposite of "touch the IP as little as possible". Option B only
runs after the visitor has demonstrably ended up with nothing.
### What B costs
- **Needs JavaScript.** A no-JS visitor sees today's behaviour. Acceptable: the
entire interactive tool already needs JS.
- **One extra round trip** after the decline, ~200 bytes each way.
- **A 204 for everyone in the fallback path when the feature is off.** The
alternative — giving the frontend its own copy of the flag to decide whether
to fetch — is two flags to keep in sync and a new way for the UI and the
server to disagree. One flag, one 204, is the better trade.
### If you overrule this and want SSR anyway
Do it as a *separate* route (`/near` or `?near=1`), not on the cached `/`, and
give that route `Cache-Control: private, no-store`. Never make the canonical,
indexable homepage location-varying — that is also an SEO hazard: a crawler in
one datacentre would index a page claiming a city, and different crawler IPs
would see different "canonical" content.
---
## 5. Privacy design
### The IP is never persisted — mechanically, not by convention
- `geoip.lookup(ip)` takes a string, memory-maps a file, returns a dict. It
writes nothing, logs nothing, and never raises.
- `api_geoip` uses **no `RunAudit`** (which would record the derived lat/lon)
and adds **no metric dimension** (the counters are aggregate by category).
- The route is excluded from the access log. `metrics.classify_inbound()` now
returns a dedicated `"geoip"` category and the middleware in `web/app.py`
skips `audit.log_access` for it. Every other route still logs, and
`test_a_normal_route_still_is_logged` guards that exclusion from widening.
This is the important one. Today the access log stores a raw IP per request.
Without the exclusion, the *one* endpoint whose whole job is turning an IP
into a place would leave a stored, timestamped, joinable (IP,
approximate-location) pair on disk and in Loki. That artefact — not the
transient lookup — is what would actually constitute keeping location data
about identifiable people. The exclusion is not defence in depth; it is the
substance of the claim.
This does **not** depend on the separate access-log IP-truncation work. That
change is still worth doing, and it composes: truncation reduces what the
*other* routes retain, while this exclusion means the geoip route retains
nothing at all either way.
- `Cache-Control: no-store, private` + `Vary: *` on the response, so it cannot
be stored or reused by any intermediary.
- The endpoint takes **no parameters**. You cannot ask it about somebody else's
address; the only input is the connection you already own. `?ip=` is ignored
(tested).
### Why no consent banner is needed — and why the analytics work is different
Under GDPR the lookup is **transient processing to deliver a service the user
explicitly asked for**. The visitor pressed "Use my location" *wanting* local
weather; when the browser can't supply it, deriving a coarse city from the
connection they are already using to reach us is a proportionate substitute for
the same, requested purpose. Nothing is stored, nothing is profiled, nothing is
shared with a third party (the database is a file we host — no processor
relationship, no international transfer, no disclosure to name in the policy).
The data subject can override the result with one tap, and the result is
explicitly labelled a guess rather than presented as a finding about them.
**ePrivacy Art. 5(3) is not engaged at all**, and this is the cleanest part of
the argument. That article governs *storing information on, or gaining access to
information stored in, a user's terminal equipment* — which is why cookies,
localStorage and device fingerprinting need consent regardless of whether the
data is personal. An IP address arrives in the request headers as an unavoidable
property of routing a packet. We neither store anything on the device nor read
anything from it. There is nothing for a consent banner to be about.
**Contrast with the analytics session-id work.** That one deliberately *writes
an identifier onto the visitor's device* in order to correlate their actions
across a session. That is squarely Art. 5(3) storage/access, it is not strictly
necessary to deliver a service the user requested, and consent is therefore
required — hence `consent.js` and the consent gate on that branch. The two
features look adjacent ("both touch visitor data") and are legally opposite:
one reads a routing header and forgets it, the other plants a durable
identifier. Keep them separate in the code and in the privacy copy, and never
let the geoip fallback be folded behind the analytics consent gate — that would
imply consent is what makes it lawful, which is the wrong basis and would
silently break the fallback for everyone who declines analytics.
### The privacy page follows the flag
`frontend/templates/privacy.html.j2` currently says, in the visitor's own
words: *"Thermograph never looks up your location from your IP address."*
Turning this feature on makes that sentence **false**, and a stale privacy
statement is a worse outcome than not shipping the feature.
So the paragraph is switched on `THERMOGRAPH_GEOIP`, which both containers read
from the same `/etc/thermograph.env` (`content.py`'s `geoip_enabled()`). The
statement and the behaviour flip in the same deploy, and the frontend unit tests
assert both branches. The frontend reads the flag for *nothing else* — it does
not gate any behaviour on it.
---
## 6. Graceful degradation
Every one of these returns `None` → HTTP 204 → the client does exactly what the
site does today. There is no failure mode worse than the status quo.
| Case | Handled where | Behaviour |
| --- | --- | --- |
| Flag unset | `enabled()` | 204 |
| Flag set, no database file | `enabled()` (`os.path.isfile`) | 204 |
| `maxminddb` not installed | lazy import in `_open()` | 204 |
| Truncated / corrupt / half-written file | `_open()` try/except | 204 (tested) |
| Private ranges (10/8, 172.16/12, 192.168/16) | `parse_ip` | 204 — every LAN-dev request |
| Loopback, link-local, multicast, reserved, unspecified | `parse_ip` | 204 |
| **CGNAT 100.64/10** | `parse_ip` + explicit `_EXTRA_RESERVED` | 204 — common on mobile carriers |
| IPv6 (global) | `parse_ip` | looked up normally; MMDB covers v6 |
| IPv6 ULA `fc00::/7`, doc `2001:db8::/32` | `_EXTRA_RESERVED` | 204 |
| IPv4-mapped IPv6 `::ffff:a.b.c.d` | `parse_ip` unwraps | treated as the v4 address |
| `1.2.3.4:5678`, `[v6]:443`, `%zone` suffixes | `parse_ip` | normalised |
| Missing `X-Forwarded-For` | `_client_ip` falls back to peer | peer is loopback behind Caddy → 204 |
| A whole XFF chain passed as one string | `parse_ip` | rejected (the route already takes the left-most hop) |
| No record for the address | `lookup()` | 204 |
| Record with a country but **no city** | `suggestion_from_record` | 204 — the VPN/proxy case |
| Record whose stated radius > 100 km | `suggestion_from_record` | 204 |
| Reader raises on a bad node | `lookup()` try/except | 204 |
The explicit `_EXTRA_RESERVED` list is not redundant with `ipaddress.is_global`:
the exact membership of `is_private`/`is_global` has moved between CPython
releases (100.64/10 in particular). Which Python the container happens to run
must not decide whether we try to geolocate a carrier-NAT address.
---
## 7. The UI — presenting a guess as a guess
Copy shipped in `frontend/static/geoip.js`:
> **Showing weather near Rotterdam, South Holland.**
> That is a rough guess from your internet connection, not from your device — it
> can be tens of kilometres out. *IP Geolocation by DB-IP*
>
> **[ Not right? Choose your spot ]**
Design rules, and why each one:
1. **Named as a guess, in the same breath as the result.** Not a footnote, not a
tooltip. City-level accuracy is ~50 km against a ~2 mile grid cell — the
product's own unit of precision is 25× finer than the input.
2. **The correction is a permanent, full-width, ≥44 px control**, not a
dismissible toast. If the guess is wrong, fixing it must be the most obvious
thing on screen.
3. **A guess is never persisted.** `updateHash()` returns early while
`approx` is true: no `localStorage` write, no URL hash. A guess that became
sticky would be indistinguishable from a choice on the next visit, and a
guessed URL would propagate a rough location the sender never picked when
shared. Only an explicit pick is remembered.
4. **The headings are rewritten while approximate.** The hero says "Roughly near
Rotterdam" instead of "Today in …", and the results heading says "Near
Rotterdam" with an "approximate — from your connection, not your device"
tag. This matters more than it sounds: the guessed cell centre gets
reverse-geocoded like any other point, which would otherwise name a
*neighbourhood* — presenting street-level precision for a city-level input.
5. **Visually recessed**`--surface-2`, dashed border, muted note text. It
reads as an aside, not as an answer.
6. **It never pre-empts the permission prompt.** It runs *after* a
declined/failed prompt, or on load only where the browser will not offer
geolocation at all (insecure context) and nothing is remembered.
7. **Never for a visitor who already has a place.** Theirs is better.
Attribution rides *with the result*, which is what CC BY 4.0 and DB-IP's own
wording ask for ("on pages displaying results") — and it is derived from the
installed file's metadata, so swapping databases changes the credit
automatically.
### Interaction with `home.locate`
`home.locate` still means exactly what it means today: "someone tapped *Use my
location*", whatever the outcome. It is unchanged, so the existing series stays
comparable across this change. Two new allowlisted events sit beside it:
- `home.geoip_shown` — a suggestion was rendered. `home.locate` minus
`home.geoip_shown` is roughly the population still hitting the dead end.
- `home.geoip_fixed` — the visitor corrected the guess. This is the accuracy
signal: a high ratio means the database is putting people in the wrong place
and the feature is doing harm, which is the number to watch after rollout.
Both are aggregate counts through the existing beacon, with **no location
dimension** — the suggestion itself is never recorded anywhere.
---
## 8. Database lifecycle
**On the host, refreshed by a systemd timer, bind-mounted read-only.** Not baked
into the image.
- `infra/deploy/geoip-refresh.sh` — resolves the current month's DB-IP release
(falling back to last month's, since the new file appears partway through the
1st), downloads, gunzips, **verifies it opens and answers for a known public
address**, and only then swaps it in atomically with `install`+`mv`. Exits 0
on "already current". A failure leaves the existing file untouched — a failed
refresh must never be worse than a stale one.
- `geoip-refresh.timer``OnCalendar=Mon 04:17 UTC`, `RandomizedDelaySec=2h`,
`Persistent=true`. Weekly rather than monthly even though DB-IP publishes
monthly: a no-op run costs one HEAD request, and it means a missed or failed
run self-heals within a week. (It also lands inside MaxMind's 30-day deletion
window without further thought, should you switch.)
- `infra/docker-compose.yml` mounts `${THERMOGRAPH_GEOIP_HOST_DIR:-/var/lib/thermograph/geoip}:/geoip:ro`
into the backend. Read-only on purpose — the app must never be able to write it.
- `geoip.py` re-opens when the file's `(mtime, size)` changes, so a refresh is
picked up **without a restart or a deploy**.
- `MODE_MMAP`: the 125 MB file costs page cache, not process RSS, and is shared
across the four uvicorn workers on a host.
**Why not build-time.** It is a 125 MB data file on its own release cadence, not
code. Baking it in adds 125 MB to every image pull, freezes freshness to
whenever the last deploy happened, ties a data refresh to a code deploy, and
(under GeoLite's EULA) cannot honour the 30-day deletion requirement on its own.
**Why not object storage.** We already have the bucket and could pull from it,
but that adds credentials and a network dependency to fetch a file that is
public and unauthenticated at the source. If a mirror is wanted later — to stop
depending on DB-IP's uptime, or to pin exactly which release is live across the
fleet — point `GEOIP_URL` at the bucket; the script needs no other change.
**Licence key storage.** With DB-IP: **none needed**, which is a large part of
the recommendation. If MaxMind is chosen instead, the key goes in the SOPS vault
(`infra/deploy/secrets/{prod,beta}.yaml`), renders into `/etc/thermograph.env`
like every other secret, and `geoip-refresh.service` already picks it up via its
`EnvironmentFile=` — set `GEOIP_URL` to the permalink that embeds it. Note the
key would then be in a *URL*, so keep it out of any log line that echoes the
URL: `geoip-refresh.sh` logs `$url`, and that line must be changed before using
a key-bearing URL. **Flagged as a real footgun if you pick MaxMind.**
---
## 9. Interaction with caching / CDN
- **The homepage stays byte-identical for every visitor** — the container is
server-rendered empty and hidden. `test_home_html_is_identical_regardless_of_client_address`
asserts the body *and* the ETag match across different `X-Forwarded-For`
values. That property is what keeps the homepage safe to cache, and it is the
main reason the lookup is not in SSR.
- **The endpoint is `no-store, private` + `Vary: *`** and answers 204 far more
often than 200. Nothing about it is shareable between visitors.
- **A CDN in front of the apex stays viable.** If Cloudflare (or a Caddy `cache`
directive) is added later, the rule is one line: never cache `/api/*`. That is
already the sane default and already how Caddy path-splits.
- **The existing frontend caches are untouched.** `api_client.py`'s TTL/LRU
cache never sees this request (it is browser→backend, not frontend→backend),
and `static/cache.js`'s IndexedDB layer is keyed by `/cell` URLs. A guessed
location produces ordinary `/api/v2/grade` traffic for an ordinary cell, so it
warms and reuses the same caches as any other place — no new cache keyspace.
- **One second-order effect worth naming:** guessed locations concentrate on
city centroids, so they will warm the *same* handful of cells repeatedly.
That is good for us (high cache hit rate, no extra Open-Meteo quota) but it
does mean per-cell request counts stop being a clean proxy for distinct users
near that cell.
---
## 10. Rollout
1. Merge with the flag off. Nothing changes — that is the point, and the tests
assert it.
2. On **beta only**: install the timer, run `geoip-refresh.sh` once by hand,
confirm `/var/lib/thermograph/geoip/city.mmdb` exists and
`curl -s -o /dev/null -w '%{http_code}' https://beta.thermograph.org/api/v2/geoip`
still returns 204 (flag still off).
3. Set `THERMOGRAPH_GEOIP=1` in beta's vault entry, redeploy **both** containers
(backend for the lookup, frontend for the privacy copy).
4. Verify by hand: decline the permission prompt; confirm the banner appears,
the correction button works, the guess is *not* in `localStorage` and *not*
in the URL; confirm `/privacy` now describes the fallback; confirm no
`"cat":"geoip"` line and no matching IP appears in the access log.
5. Watch `home.geoip_fixed / home.geoip_shown` for a week. If a large share of
visitors are correcting the guess, the database is putting people in the
wrong place — turn it off rather than tuning copy.
6. Only then, prod.
Rollback is `THERMOGRAPH_GEOIP=0` and a redeploy; or just delete the database
file, which disables it without touching config.
---
## 11. Decisions needed from the operator
1. **Which database?** Recommendation: **DB-IP IP to City Lite** (no account, no
key, no retention clock, CC BY 4.0). The alternative is GeoLite2 City for
`accuracy_radius` and weekly updates, at the cost of a vault-managed licence
key and a 30-day deletion obligation. Code supports either unchanged.
2. **Attribution placement.** Currently in the suggestion banner itself, beside
the result. Alternatives: also on `/about`, or only on `/about`. DB-IP's
stated wording asks for it "on pages displaying results", which the banner
satisfies — confirm you're happy with a small credit line inside the hero.
3. **SSR or client-side?** Recommendation: **client-side** (§4). Overruling this
means accepting a location-varying homepage; if you do, put it on a separate
non-canonical route, never on cached `/`.
4. **Refresh cadence.** Currently weekly-with-no-op (§8). Alternatives: monthly
to match DB-IP's release cadence exactly, or daily. Weekly is the
self-healing middle.
5. **Auto-suggest in insecure contexts?** Today the prototype suggests on load
when the browser won't offer geolocation at all (LAN dev over plain HTTP) and
nothing is remembered, since there is no prompt to decline. That is the only
path where a lookup happens without an explicit user action. Prod is HTTPS so
it never triggers there — but say if you'd rather it *never* happen without a
tap, and it becomes a one-line change.
6. **Threshold for calling it off.** Pick the `home.geoip_fixed / home.geoip_shown`
ratio at which the feature gets turned off, *before* rollout rather than
after — otherwise the number will always look explainable.
---
## 12. Things I think are bad ideas — flagged, not implemented
- **Don't put the lookup in the SSR homepage.** §4. A location-varying
canonical page is a cache-poisoning and SEO hazard, and it geolocates the
majority of visitors who already have a saved place and will never use it.
- **Don't persist the guess.** Not to `localStorage`, not to the URL, not to an
account preference. It is not a choice and must never become one silently.
The branch enforces this in `updateHash()`.
- **Don't use a third-party geolocation API**, even a "privacy-friendly" one.
Sending visitor IPs to another company creates a processor relationship and a
disclosure obligation, for a file we can just host. This was already the
brief's requirement; restating it because it is the single most tempting
shortcut here.
- **Don't extend this to country-level personalisation** (units, language,
default city) on the same lookup. Unit selection already works off the picked
location's country code, and quietly widening an IP lookup from "a fallback
the user asked for" to "we adapt the site to you" changes the legal analysis
in §5 — it stops being strictly necessary for a requested service.
- **Don't reuse the derived city anywhere it could be stored.** No metrics
dimension, no audit field, no "popular cities" table fed from guesses. The
moment a derived location is written next to anything timestamped, the
privacy claim in §5 stops being true.
- **Don't fold this behind the analytics consent gate.** §5. Different legal
basis; gating it on consent both implies the wrong basis and breaks the
fallback for everyone who declines analytics.
- **Don't log `$url` once a MaxMind key-bearing permalink is in use** (§8).
- **Be honest that this is a downgrade, not a feature.** A city-level guess is
25× coarser than the grid the whole product is built on. It exists so a
visitor who declined isn't stranded — not because IP location is good. If the
correction rate says people don't want it, delete it rather than defend it.

View file

@ -118,6 +118,11 @@ def classify_inbound(path: str, base: str = "") -> str:
# category keeps record_inbound from double-counting every interaction.
if seg == "event":
return "event"
# Its own category so web/app.py's access-log middleware can skip it:
# that route turns the client IP into a coarse location, so an access
# line pairing the two on disk is exactly what the feature avoids.
if seg == "geoip":
return "geoip"
return f"api:{seg}" if seg else "api:other"
if p.endswith((".js", ".css", ".html", ".webmanifest", ".png", ".svg", ".ico",
".json", ".woff", ".woff2", ".map", ".txt", ".xml")):
@ -141,6 +146,13 @@ def classify_inbound(path: str, base: str = "") -> str:
EVENTS = frozenset({
"home.locate", "home.digest_signup", "home.records_click", "home.share",
# Approximate-location fallback (data/geoip.py). Pairs with home.locate:
# that one counts taps on "Use my location" whatever the outcome, these two
# count how often the IP fallback rescued a visitor who got no position
# (geoip_shown) and how often the guess was wrong enough to correct
# (geoip_fixed). Aggregate counts only — no location dimension, so the
# suggestion itself is never recorded anywhere.
"home.geoip_shown", "home.geoip_fixed",
})
# home.nav_{surface}: only these surfaces, so the key space stays bounded.
NAV_SURFACES = frozenset({

330
backend/data/geoip.py Normal file
View file

@ -0,0 +1,330 @@
"""Coarse IP -> approximate-location lookup, for visitors who don't share
browser geolocation.
This is a *fallback suggestion*, never a position. Browser geolocation is
metres-accurate; a city-level IP database is tens of kilometres accurate against
a ~2 mile grid cell, so everything here is shaped so the caller cannot mistake
one for the other: the payload is explicitly labelled approximate, and the UI
that consumes it must offer a correction affordance (see frontend/static/geoip.js).
Design rules this module enforces, in order of importance:
1. **The IP is never persisted.** ``lookup()`` takes a string, reads a memory
-mapped database, and returns a dict. Nothing is written anywhere no DB
row, no log line, no metric dimension. The web route deliberately keeps the
geoip category out of the access log too (see web/app.py), so there is never
a joinable (IP, derived-city) record on disk. Callers must not log the
argument.
2. **The lookup is local.** The database is a file on our own disk in MaxMind's
open MMDB format. No third-party API is called, so the visitor's IP never
leaves the server and no data-processor relationship is created.
3. **Off unless deliberately turned on.** ``THERMOGRAPH_GEOIP`` must be truthy
*and* ``THERMOGRAPH_GEOIP_DB`` must point at a readable file *and* the
``maxminddb`` reader must be importable. Any one of those missing makes
``enabled()`` False and every ``lookup()`` return None which the route
turns into a 204 and the client treats exactly like today's no-fallback
behaviour. There is no failure mode here that is worse than the status quo.
4. **Silence beats a wrong guess.** A reserved/private/CGNAT/loopback address,
an unparseable one, a record with no city (a country centroid would put a
VPN user in the geographic middle of a country they aren't in), or a record
whose stated accuracy radius is worse than
``THERMOGRAPH_GEOIP_MAX_RADIUS_KM`` all return None rather than something
plausible-looking.
Database choice is a deployment concern, not a code one: both DB-IP's
*IP to City Lite* (CC BY 4.0, no account) and MaxMind's *GeoLite2 City*
(GeoLite EULA, account + licence key) ship the same MMDB format with the same
record shape, so either file works unchanged. ``attribution()`` reads the
file's own metadata so the credit line the UI renders follows whichever file is
installed. See GEOIP-APPROX-LOCATION.md for the comparison and the recommendation.
"""
import ipaddress
import os
import threading
from data import grid
FLAG_ENV = "THERMOGRAPH_GEOIP"
DB_ENV = "THERMOGRAPH_GEOIP_DB"
RADIUS_ENV = "THERMOGRAPH_GEOIP_MAX_RADIUS_KM"
ATTRIB_ENV = "THERMOGRAPH_GEOIP_ATTRIBUTION"
# City-level databases claim ~50 km typical accuracy. Anything the file itself
# admits is worse than this is a region/country centroid dressed up as a city,
# which is exactly the "wrong part of the map" guess we'd rather not make.
DEFAULT_MAX_RADIUS_KM = 100.0
# Extra reserved ranges checked explicitly rather than leaning on
# ipaddress.is_global alone: the exact membership of is_private/is_global has
# moved between CPython releases (100.64/10 in particular), and "which Python
# is this container on" must not decide whether we try to geolocate a
# carrier-NAT address.
_EXTRA_RESERVED = (
ipaddress.ip_network("100.64.0.0/10"), # RFC 6598 carrier-grade NAT
ipaddress.ip_network("192.0.0.0/24"), # IETF protocol assignments
ipaddress.ip_network("198.18.0.0/15"), # benchmarking
ipaddress.ip_network("fc00::/7"), # IPv6 unique-local
ipaddress.ip_network("2001:db8::/32"), # documentation
)
_lock = threading.Lock()
_reader = None
_reader_key: "tuple | None" = None # (path, mtime, size) the open reader came from
def _truthy(v: "str | None") -> bool:
return (v or "").strip().lower() in ("1", "true", "yes", "on")
def db_path() -> str:
return os.environ.get(DB_ENV, "").strip()
def max_radius_km() -> float:
try:
return float(os.environ.get(RADIUS_ENV, "") or DEFAULT_MAX_RADIUS_KM)
except ValueError:
return DEFAULT_MAX_RADIUS_KM
def enabled() -> bool:
"""True only when the flag is set AND a database file is actually present.
Read on every call (not cached) so flipping the flag or dropping the file in
doesn't need a restart, and so a test can toggle it with monkeypatch.setenv."""
if not _truthy(os.environ.get(FLAG_ENV)):
return False
path = db_path()
return bool(path) and os.path.isfile(path)
def _open():
"""The memory-mapped reader for the configured database, or None.
Re-opens when the file's (mtime, size) changes so the monthly refresh job
can swap the file in atomically and be picked up without a deploy. mmap
means the 100-200 MB file costs page cache, not process RSS, and is shared
across uvicorn workers on the same host.
"""
global _reader, _reader_key
path = db_path()
if not path:
return None
try:
st = os.stat(path)
except OSError:
return None
key = (path, st.st_mtime_ns, st.st_size)
with _lock:
if _reader is not None and _reader_key == key:
return _reader
try:
import maxminddb
except ImportError:
return None
try:
new = maxminddb.open_database(path, maxminddb.MODE_MMAP)
except Exception: # noqa: BLE001 - a corrupt/half-written file must not 500
return None
old = _reader
_reader, _reader_key = new, key
if old is not None:
try:
old.close()
except Exception: # noqa: BLE001
pass
return new
def reset() -> None:
"""Drop the cached reader. For tests, and for anything that rewrites the
database in place rather than swapping it atomically."""
global _reader, _reader_key
with _lock:
r, _reader, _reader_key = _reader, None, None
if r is not None:
try:
r.close()
except Exception: # noqa: BLE001
pass
def parse_ip(raw: "str | None") -> "ipaddress._BaseAddress | None":
"""The routable address in ``raw``, or None.
Handles the shapes an X-Forwarded-For hop actually arrives in a bare
address, ``[v6]:port``, ``v4:port``, a ``%zone`` suffix and rejects
everything that can't identify a location on the public internet: private
LAN ranges (every LAN-dev and internal-proxy request), loopback, link-local,
multicast, carrier-grade NAT, IPv6 unique-local, and anything unparseable.
An IPv4-mapped IPv6 address is unwrapped so ``::ffff:8.8.8.8`` is treated as
the IPv4 address it is.
"""
s = (raw or "").strip()
if not s:
return None
if s.startswith("["): # [2001:db8::1]:443
s = s[1:].partition("]")[0]
elif s.count(":") == 1: # 8.8.8.8:443 (a lone colon is never v6)
s = s.partition(":")[0]
s = s.partition("%")[0] # fe80::1%eth0
try:
addr = ipaddress.ip_address(s)
except ValueError:
return None
mapped = getattr(addr, "ipv4_mapped", None)
if mapped is not None:
addr = mapped
if not addr.is_global:
return None
if addr.is_private or addr.is_loopback or addr.is_link_local:
return None
if addr.is_multicast or addr.is_reserved or addr.is_unspecified:
return None
for net in _EXTRA_RESERVED:
if addr.version == net.version and addr in net:
return None
return addr
def _name(node) -> str:
"""The English display name out of an MMDB names map. Both DB-IP Lite and
GeoLite2 store ``{"names": {"en": ..., "de": ...}}``; DB-IP Lite is
English-only in some builds, so fall back to whatever single name exists
rather than dropping an otherwise-good record."""
if not isinstance(node, dict):
return ""
names = node.get("names")
if not isinstance(names, dict):
return ""
for key in ("en", "en-US"):
v = names.get(key)
if isinstance(v, str) and v.strip():
return v.strip()
for v in names.values():
if isinstance(v, str) and v.strip():
return v.strip()
return ""
def suggestion_from_record(record, *, max_radius: "float | None" = None) -> "dict | None":
"""Turn one raw MMDB record into the wire payload, or None if it is too
coarse to be worth showing. Split out from ``lookup()`` so every rejection
rule is testable without a database file."""
if not isinstance(record, dict):
return None
loc = record.get("location")
if not isinstance(loc, dict):
return None
lat, lon = loc.get("latitude"), loc.get("longitude")
if not isinstance(lat, (int, float)) or not isinstance(lon, (int, float)):
return None
if not (-90.0 <= lat <= 90.0) or not (-180.0 <= lon <= 180.0):
return None
# No city name means a country (or continent) centroid. Showing one would
# put every VPN user in the geographic middle of a country they may not be
# in, which is worse than showing nothing: it looks like a real answer.
city = _name(record.get("city"))
if not city:
return None
radius = loc.get("accuracy_radius")
limit = max_radius_km() if max_radius is None else max_radius
if isinstance(radius, (int, float)) and radius > limit:
return None
subs = record.get("subdivisions")
region = _name(subs[0]) if isinstance(subs, list) and subs else ""
country_node = record.get("country") or record.get("registered_country") or {}
country = _name(country_node)
code = country_node.get("iso_code") if isinstance(country_node, dict) else None
# Snap to the app's own ~2 mile grid. This drops the database's fake
# precision (city centroids are published to 4+ decimal places, which is
# metres, for a value good to tens of kilometres) and returns exactly the
# cell the weather would be fetched for anyway — so the coordinate carries
# no more information than the city name already did.
cell = grid.snap(float(lat), float(lon))
# "Springfield" alone is ambiguous; "Springfield, Illinois" is not. Region
# first (more useful locally), country when there is no region.
label = f"{city}, {region}" if region else (f"{city}, {country}" if country else city)
return {
"approximate": True, # never omit: the client keys its copy off this
"source": "ip",
"label": label,
"city": city,
"region": region or None,
"country": country or None,
"country_code": code if isinstance(code, str) else None,
"lat": cell["center_lat"],
"lon": cell["center_lon"],
"accuracy_radius_km": radius if isinstance(radius, (int, float)) else None,
"attribution": attribution(),
}
def lookup(ip: "str | None") -> "dict | None":
"""An approximate-location suggestion for ``ip``, or None.
None covers every degraded case feature off, database missing, private or
malformed address, no matching record, record too coarse so the caller has
exactly one branch to write and it is always "behave as if this feature did
not exist". Never raises; never logs or stores ``ip``.
"""
if not enabled():
return None
addr = parse_ip(ip)
if addr is None:
return None
reader = _open()
if reader is None:
return None
try:
record = reader.get(str(addr))
except Exception: # noqa: BLE001 - a bad record must degrade, not 500
return None
if record is None:
return None
return suggestion_from_record(record)
# --- attribution -----------------------------------------------------------
# Both candidate databases are free *with attribution*, and each wants its own
# wording, so the credit is derived from the installed file's own metadata
# instead of hardcoded — swapping databases stays a config change. An explicit
# THERMOGRAPH_GEOIP_ATTRIBUTION ("text|url") overrides, for a file whose
# database_type we don't recognise.
_ATTRIBUTIONS = {
"dbip": ("IP Geolocation by DB-IP", "https://db-ip.com"),
"geolite2": ("This product includes GeoLite data created by MaxMind",
"https://www.maxmind.com"),
}
def attribution() -> "dict | None":
"""``{"text", "url"}`` for the installed database, or None if it can't be
determined. Rendered next to any result the visitor sees both licences
require credit on pages that display their data."""
override = os.environ.get(ATTRIB_ENV, "").strip()
if override:
text, _, url = override.partition("|")
return {"text": text.strip(), "url": url.strip() or None}
reader = _open()
if reader is None:
return None
try:
kind = (reader.metadata().database_type or "").lower()
except Exception: # noqa: BLE001
return None
if "dbip" in kind or "db-ip" in kind:
text, url = _ATTRIBUTIONS["dbip"]
elif "geolite" in kind:
text, url = _ATTRIBUTIONS["geolite2"]
else:
return None
return {"text": text, "url": url}

View file

@ -25,6 +25,13 @@ websockets==16.0
# Recurring worker-tier jobs — city warming, IndexNow pings (see
# notifications/scheduler.py). In-process; no broker/queue needed at this scale.
apscheduler==3.11.3
# MMDB reader for the optional approximate-location fallback (data/geoip.py).
# The reader only, not MaxMind's geoip2 model layer — the raw dict record is all
# we read, and staying on maxminddb keeps DB-IP's and MaxMind's files equally
# usable. Imported lazily inside geoip.py so a missing wheel disables the
# feature instead of breaking boot; pinned here so the flag is flippable on a
# deployed image without a rebuild.
maxminddb==2.8.2
# The lake role's SQL engine (lake_app.py /query): embedded, parallel
# partition-pruned scans over the ERA5 hive table. Only lake_app imports it.
duckdb==1.4.2

View file

@ -0,0 +1,263 @@
"""Approximate-location fallback (data/geoip.py).
Every rejection rule is tested against a raw MMDB-shaped record rather than a
real database file, so the suite stays hermetic and needs no 125 MB download
the record shape below is what both DB-IP IP-to-City Lite and MaxMind GeoLite2
City actually return.
"""
import os
import pytest
from data import geoip
from data import grid
def _record(**over) -> dict:
"""A well-formed city record. Rotterdam, as in the design doc's example."""
rec = {
"city": {"names": {"en": "Rotterdam"}},
"country": {"iso_code": "NL", "names": {"en": "Netherlands"}},
"subdivisions": [{"names": {"en": "South Holland"}}],
"location": {"latitude": 51.9225, "longitude": 4.47917, "accuracy_radius": 20},
}
rec.update(over)
return rec
# --- address parsing --------------------------------------------------------
@pytest.mark.parametrize("raw", [
"8.8.8.8",
"8.8.8.8:51234", # an XFF hop can carry the source port
"2001:4860:4860::8888",
"[2001:4860:4860::8888]:443",
"::ffff:8.8.8.8", # IPv4-mapped IPv6 unwraps to the v4 address
" 8.8.8.8 ",
])
def test_parse_accepts_routable_addresses(raw):
assert geoip.parse_ip(raw) is not None
@pytest.mark.parametrize("raw", [
None, "", " ", "not-an-ip", "999.1.1.1", "8.8.8.8, 9.9.9.9", # a whole XFF chain
"127.0.0.1", "::1", # loopback
"10.0.0.4", "192.168.1.20", "172.16.5.5", # RFC 1918 — every LAN-dev request
"100.64.7.9", # RFC 6598 carrier-grade NAT (mobile networks)
"169.254.10.1", "fe80::1%eth0", # link-local
"fd00::1", # IPv6 unique-local
"2001:db8::1", # documentation range
"224.0.0.1", # multicast
"0.0.0.0", "::", # unspecified
])
def test_parse_rejects_everything_unroutable(raw):
assert geoip.parse_ip(raw) is None
def test_ipv4_mapped_v6_resolves_to_the_v4_address():
assert str(geoip.parse_ip("::ffff:8.8.8.8")) == "8.8.8.8"
# --- record -> suggestion ---------------------------------------------------
def test_good_record_becomes_a_labelled_approximate_suggestion():
s = geoip.suggestion_from_record(_record())
assert s["approximate"] is True and s["source"] == "ip"
assert s["label"] == "Rotterdam, South Holland"
assert s["city"] == "Rotterdam" and s["country_code"] == "NL"
def test_coordinates_are_snapped_to_the_app_grid_not_passed_through():
"""The database publishes a city centroid to five decimal places — metre
precision for a value good to tens of kilometres. We return the grid cell
centre instead, so the payload carries no more precision than it has
information."""
s = geoip.suggestion_from_record(_record())
cell = grid.snap(51.9225, 4.47917)
assert (s["lat"], s["lon"]) == (cell["center_lat"], cell["center_lon"])
assert s["lat"] != 51.9225
def test_region_absent_falls_back_to_country_in_the_label():
s = geoip.suggestion_from_record(_record(subdivisions=[]))
assert s["label"] == "Rotterdam, Netherlands"
def test_city_only_record_still_labels():
s = geoip.suggestion_from_record(
{"city": {"names": {"en": "Rotterdam"}}, "location": {"latitude": 1.0, "longitude": 2.0}})
assert s["label"] == "Rotterdam"
def test_non_english_only_names_are_still_usable():
s = geoip.suggestion_from_record(_record(city={"names": {"nl": "Rotterdam"}}))
assert s["city"] == "Rotterdam"
@pytest.mark.parametrize("rec", [
None, {}, "garbage",
{"location": {"latitude": 51.9, "longitude": 4.4}}, # country-only: no city
{"city": {"names": {"en": "X"}}}, # no coordinates
{"city": {"names": {"en": "X"}}, "location": {"latitude": 51.9}}, # half coordinates
{"city": {"names": {"en": "X"}},
"location": {"latitude": "51.9", "longitude": "4.4"}}, # wrong types
{"city": {"names": {"en": "X"}},
"location": {"latitude": 991.0, "longitude": 4.4}}, # out of range
])
def test_unusable_records_suggest_nothing(rec):
assert geoip.suggestion_from_record(rec) is None
def test_a_country_centroid_is_never_offered_as_a_city():
"""The single most damaging failure mode: a VPN/proxy user gets a record
with a country but no city, and we drop a pin in the middle of a country
they are not in. Silence is the correct answer."""
rec = _record()
del rec["city"]
assert geoip.suggestion_from_record(rec) is None
def test_record_wider_than_the_accuracy_limit_is_rejected():
assert geoip.suggestion_from_record(
_record(location={"latitude": 51.9, "longitude": 4.4, "accuracy_radius": 500})) is None
def test_record_with_no_stated_accuracy_is_accepted():
"""DB-IP Lite omits accuracy_radius entirely; requiring it would disable the
feature for that database."""
s = geoip.suggestion_from_record(
_record(location={"latitude": 51.9, "longitude": 4.4}))
assert s is not None and s["accuracy_radius_km"] is None
def test_accuracy_limit_is_configurable(monkeypatch):
monkeypatch.setenv(geoip.RADIUS_ENV, "1000")
assert geoip.suggestion_from_record(
_record(location={"latitude": 51.9, "longitude": 4.4, "accuracy_radius": 500})) is not None
# --- feature flag -----------------------------------------------------------
def test_disabled_by_default(monkeypatch):
monkeypatch.delenv(geoip.FLAG_ENV, raising=False)
monkeypatch.delenv(geoip.DB_ENV, raising=False)
assert geoip.enabled() is False
assert geoip.lookup("8.8.8.8") is None
def test_flag_without_a_database_stays_disabled(monkeypatch, tmp_path):
monkeypatch.setenv(geoip.FLAG_ENV, "1")
monkeypatch.setenv(geoip.DB_ENV, str(tmp_path / "missing.mmdb"))
assert geoip.enabled() is False
assert geoip.lookup("8.8.8.8") is None
def test_database_without_the_flag_stays_disabled(monkeypatch, tmp_path):
db = tmp_path / "city.mmdb"
db.write_bytes(b"not really an mmdb")
monkeypatch.delenv(geoip.FLAG_ENV, raising=False)
monkeypatch.setenv(geoip.DB_ENV, str(db))
assert geoip.enabled() is False
def test_unreadable_database_degrades_to_no_suggestion(monkeypatch, tmp_path):
"""A truncated or half-written file (a refresh job interrupted mid-copy)
must behave like the feature is off, not raise."""
db = tmp_path / "city.mmdb"
db.write_bytes(b"not really an mmdb")
monkeypatch.setenv(geoip.FLAG_ENV, "1")
monkeypatch.setenv(geoip.DB_ENV, str(db))
geoip.reset()
assert geoip.enabled() is True
assert geoip.lookup("8.8.8.8") is None
geoip.reset()
# --- lookup, end to end over a stand-in reader ------------------------------
class _FakeReader:
"""Stands in for maxminddb's reader: same tiny surface geoip.py uses."""
def __init__(self, by_ip, database_type="DBIP-City-Lite (compat=City)"):
self.by_ip = by_ip
self.asked = []
self._type = database_type
def get(self, ip):
self.asked.append(ip)
return self.by_ip.get(ip)
def metadata(self):
return type("M", (), {"database_type": self._type})()
def close(self):
pass
@pytest.fixture
def live(monkeypatch, tmp_path):
"""Feature on, with a stand-in reader — exercises enabled() + parse_ip() +
suggestion_from_record() together without a real database file."""
db = tmp_path / "city.mmdb"
db.write_bytes(b"x")
monkeypatch.setenv(geoip.FLAG_ENV, "1")
monkeypatch.setenv(geoip.DB_ENV, str(db))
reader = _FakeReader({"8.8.8.8": _record()})
monkeypatch.setattr(geoip, "_open", lambda: reader)
return reader
def test_lookup_end_to_end(live):
s = geoip.lookup("8.8.8.8")
assert s["city"] == "Rotterdam"
assert s["attribution"] == {"text": "IP Geolocation by DB-IP", "url": "https://db-ip.com"}
def test_lookup_never_queries_the_database_for_a_private_address(live):
assert geoip.lookup("192.168.1.20") is None
assert live.asked == [] # rejected before the file is touched at all
def test_lookup_returns_none_for_an_ip_with_no_record(live):
assert geoip.lookup("9.9.9.9") is None
def test_lookup_survives_a_reader_that_raises(live, monkeypatch):
def boom(ip):
raise ValueError("corrupt node")
monkeypatch.setattr(live, "get", boom)
assert geoip.lookup("8.8.8.8") is None
def test_attribution_follows_the_installed_database(monkeypatch, tmp_path):
db = tmp_path / "city.mmdb"
db.write_bytes(b"x")
monkeypatch.setenv(geoip.FLAG_ENV, "1")
monkeypatch.setenv(geoip.DB_ENV, str(db))
monkeypatch.setattr(geoip, "_open", lambda: _FakeReader({}, "GeoLite2-City"))
assert "MaxMind" in geoip.attribution()["text"]
def test_attribution_can_be_overridden_by_env(monkeypatch):
monkeypatch.setenv(geoip.ATTRIB_ENV, "Data by Someone|https://example.org")
assert geoip.attribution() == {"text": "Data by Someone", "url": "https://example.org"}
def test_attribution_is_none_for_an_unrecognised_database(monkeypatch, tmp_path):
db = tmp_path / "city.mmdb"
db.write_bytes(b"x")
monkeypatch.delenv(geoip.ATTRIB_ENV, raising=False)
monkeypatch.setenv(geoip.DB_ENV, str(db))
monkeypatch.setattr(geoip, "_open", lambda: _FakeReader({}, "Homegrown-City"))
assert geoip.attribution() is None
@pytest.mark.skipif(os.environ.get("THERMOGRAPH_GEOIP_DB", "") == "",
reason="no real MMDB installed; set THERMOGRAPH_GEOIP_DB to run")
def test_real_database_opens_and_answers():
"""Opt-in smoke test against an actual installed database — the one thing
the stand-in reader cannot prove (that our record-shape assumptions match a
real file). Skipped in CI, run by hand after the refresh job first lands a
file on a box."""
import os as _os
_os.environ[geoip.FLAG_ENV] = "1"
geoip.reset()
assert geoip.enabled()
assert geoip.lookup("8.8.8.8") is not None or geoip.lookup("1.1.1.1") is not None
geoip.reset()

View file

@ -0,0 +1,114 @@
"""GET /api/v2/geoip — the approximate-location fallback endpoint.
The two properties worth locking down are behavioural, not cosmetic: it is
inert when the flag is off (so shipping it disabled changes nothing), and it
never writes the client IP anywhere.
"""
import glob
import json
import os
import pytest
from fastapi.testclient import TestClient
from core import audit
from data import geoip
from web import app as appmod
URL = "/thermograph/api/v2/geoip"
_SUGGESTION = {
"approximate": True, "source": "ip", "label": "Rotterdam, South Holland",
"city": "Rotterdam", "region": "South Holland", "country": "Netherlands",
"country_code": "NL", "lat": 51.92028, "lon": 4.48339,
"accuracy_radius_km": 20,
"attribution": {"text": "IP Geolocation by DB-IP", "url": "https://db-ip.com"},
}
@pytest.fixture
def client():
return TestClient(appmod.app)
def _access_lines():
lines = []
for path in glob.glob(os.path.join(audit.ACCESS_DIR, "access-*.jsonl")):
with open(path, encoding="utf-8") as fh:
lines += [json.loads(ln) for ln in fh if ln.strip()]
return lines
def test_disabled_answers_204_with_no_body(client, monkeypatch):
monkeypatch.delenv(geoip.FLAG_ENV, raising=False)
r = client.get(URL, headers={"X-Forwarded-For": "8.8.8.8"})
assert r.status_code == 204
assert r.content == b""
def test_enabled_hit_returns_the_suggestion(client, monkeypatch):
monkeypatch.setattr(geoip, "lookup", lambda ip: dict(_SUGGESTION))
r = client.get(URL, headers={"X-Forwarded-For": "8.8.8.8"})
assert r.status_code == 200
body = r.json()
assert body["approximate"] is True and body["label"] == "Rotterdam, South Holland"
assert body["attribution"]["url"] == "https://db-ip.com"
def test_response_is_never_cacheable_by_a_shared_cache(client, monkeypatch):
"""This response is per-visitor by construction. A CDN or proxy that reused
it would show one visitor another's city — the exact hazard that keeps this
lookup out of the server-rendered homepage."""
monkeypatch.setattr(geoip, "lookup", lambda ip: dict(_SUGGESTION))
r = client.get(URL, headers={"X-Forwarded-For": "8.8.8.8"})
assert "no-store" in r.headers["cache-control"]
assert r.headers["vary"] == "*"
def test_miss_answers_204(client, monkeypatch):
monkeypatch.setattr(geoip, "lookup", lambda ip: None)
r = client.get(URL, headers={"X-Forwarded-For": "8.8.8.8"})
assert r.status_code == 204
assert "no-store" in r.headers["cache-control"]
def test_the_left_most_forwarded_hop_is_what_gets_looked_up(client, monkeypatch):
seen = []
monkeypatch.setattr(geoip, "lookup", lambda ip: seen.append(ip) or None)
client.get(URL, headers={"X-Forwarded-For": "8.8.8.8, 10.0.0.1, 10.0.0.2"})
assert seen == ["8.8.8.8"]
def test_missing_forwarded_header_still_answers_cleanly(client, monkeypatch):
"""No XFF (direct hit, LAN dev) falls through to the peer address, which is
the loopback TestClient address private, so no suggestion, no error."""
monkeypatch.setenv(geoip.FLAG_ENV, "1")
r = client.get(URL)
assert r.status_code == 204
def test_the_client_ip_is_not_written_to_the_access_log(client, monkeypatch):
"""The whole point: reading an IP to derive a location must not leave a
stored, joinable (IP, location) pair behind."""
monkeypatch.setattr(geoip, "lookup", lambda ip: dict(_SUGGESTION))
before = len(_access_lines())
client.get(URL, headers={"X-Forwarded-For": "8.8.8.8"})
after = _access_lines()
assert len(after) == before
assert not any(ln.get("ip") == "8.8.8.8" for ln in after)
def test_a_normal_route_still_is_logged(client):
"""Guards the exclusion above from silently widening to everything."""
before = len(_access_lines())
client.get("/thermograph/api/version")
assert len(_access_lines()) > before
def test_the_route_takes_no_parameters(client, monkeypatch):
"""You cannot ask this endpoint about someone else's address: there is no
input but the connection itself, and a supplied one is ignored."""
seen = []
monkeypatch.setattr(geoip, "lookup", lambda ip: seen.append(ip) or None)
client.get(f"{URL}?ip=1.2.3.4", headers={"X-Forwarded-For": "8.8.8.8"})
assert seen == ["8.8.8.8"]

View file

@ -21,6 +21,7 @@ from accounts import api_accounts
from api import content_routes
from core import audit
from data import climate
from data import geoip
from notifications import digest
from notifications import discord_bot
from notifications import discord_interactions as discord_interactions_mod
@ -306,7 +307,11 @@ async def revalidate_static(request, call_next):
await run_in_threadpool(metrics.record_inbound, cat, response.status_code)
# Retain per-request client IPs for later analysis; skip static assets and the
# dashboard's own metrics polling to keep the log to real, meaningful traffic.
if cat not in ("static", "metrics", "event"):
# "geoip" is excluded for a different reason: that route derives a coarse
# location FROM the client IP, so logging the two together would create the
# one artefact this feature exists to avoid — a stored, joinable
# (IP, approximate-location) pair. See api_geoip.
if cat not in ("static", "metrics", "event", "geoip"):
await run_in_threadpool(audit.log_access, {
"ip": _client_ip(request), "method": request.method,
"path": path, "status": response.status_code, "cat": cat})
@ -809,6 +814,39 @@ async def api_event(request: Request) -> Response:
return Response(status_code=204)
async def api_geoip(request: Request) -> Response:
"""An approximate location suggested from this request's own IP, for a
visitor who has not shared browser geolocation. Feature-flagged OFF.
204 (no body) is the normal answer, and means "no suggestion" for every
reason there can be: the flag is off, no database is installed, the address
is private/reserved/unparseable, no record matched, or the record was too
coarse to be worth showing. The client treats 204 exactly as it behaves
today, so a disabled or broken lookup is indistinguishable from the feature
never having shipped.
Privacy posture, deliberately:
* the IP is read from this request's headers, used, and dropped — it is
not written to the access log (see the middleware's category skip), any
database, any metric dimension, or any log line here;
* ``no-store`` + ``Vary: *`` so no shared cache or CDN can ever hand one
visitor's suggestion to another — this is why the lookup is a separate,
client-called endpoint rather than folded into the server-rendered
homepage, whose HTML must stay byte-identical for everyone;
* GET with no parameters and no body: there is nothing a caller can pass
that would let them geolocate an address other than their own.
"""
# No RunAudit and no metrics dimension: both would record the derived
# location, which is the sensitive half of the pair.
hit = await run_in_threadpool(geoip.lookup, _client_ip(request))
if hit is None:
return Response(status_code=204, headers={"Cache-Control": "no-store"})
return Response(
content=json.dumps(hit), media_type="application/json",
headers={"Cache-Control": "no-store, private", "Vary": "*"},
)
# --- API versioning --------------------------------------------------------
# Non-backward-compatible additions land in a new version; every version stays
# mounted and served simultaneously. v1 is the original contract (grade/geocode);
@ -828,6 +866,7 @@ v2.add_api_route("/day", api_day, methods=["GET"])
v2.add_api_route("/score", api_score, methods=["GET"])
v2.add_api_route("/forecast", api_forecast, methods=["GET"])
v2.add_api_route("/cell", api_cell, methods=["GET"])
v2.add_api_route("/geoip", api_geoip, methods=["GET"], include_in_schema=False)
v2.add_api_route("/metrics", api_metrics, methods=["GET"], include_in_schema=False)
v2.add_api_route("/event", api_event, methods=["POST"], include_in_schema=False)

View file

@ -380,12 +380,27 @@ def about_page(request: Request) -> Response:
return _respond_html(request, "about.html.j2", **ctx)
def geoip_enabled() -> bool:
"""Whether the backend's approximate-location fallback is switched on
(backend/data/geoip.py). Read here for ONE reason: the privacy page's
"Your location" section makes a factual claim about IP lookups, and a
factual claim must not survive the behaviour it describes. Both containers
read the same THERMOGRAPH_GEOIP out of the same /etc/thermograph.env, so
the statement and the behaviour flip together in one deploy.
Nothing else in this repo branches on it the interactive tool just calls
the endpoint and gets a 204 when the feature is off.
"""
return os.environ.get("THERMOGRAPH_GEOIP", "").strip().lower() in ("1", "true", "yes", "on")
def privacy_page(request: Request) -> Response:
ctx = {
"canonical_path": "/privacy",
"breadcrumb": [("Home", f"{BASE}/"), ("Privacy", None)],
"page_title": PAGES["privacy"]["title"],
"page_description": PAGES["privacy"]["description"],
"geoip": geoip_enabled(),
}
return _respond_html(request, "privacy.html.j2", **ctx)

View file

@ -7,10 +7,16 @@ import { initFindButton, setFindLabel } from "./mappicker.js";
import { W, PW, H, PL, PR, plotTop, plotBot, xLabY, setChartWidth, chartPalette,
tempChart, precipChart, dryChart, attachChartHover } from "./chart.js";
import { track } from "./digest.js";
import { fetchApprox, renderApprox, clearApprox } from "./geoip.js";
import { TIER_COLORS, SCALE_TEMP, drynessColor, pctOrd, esc, todayISO, placeLabel,
tierKeySegs, GUIDE_LINK, fmtPrecip, fmtWind, fmtHumid } from "./shared.js";
let selected = null; // {lat, lon}
// True while `selected` came from the IP fallback rather than a real choice.
// A guess is loaded (so the page is useful) but never remembered and never
// written into the URL — see geoip.js for why that distinction matters.
let approx = false;
let approxLabel = ""; // the city the guess named, for headlines that must not overstate it
const dateInput = document.getElementById("date-input");
const todayBtn = document.getElementById("today-btn");
@ -32,8 +38,13 @@ todayBtn.addEventListener("click", () => {
if (selected) runGrade();
});
function selectLocation(lat, lon) {
function selectLocation(lat, lon, opts) {
selected = { lat, lon };
// An approximate pick keeps its banner up; any real pick retires it, so the
// "this is a guess" claim can never outlive the guess.
approx = !!(opts && opts.approx);
approxLabel = approx ? ((opts && opts.label) || "") : "";
if (!approx) clearApprox(geoBox);
// Show the raw coordinates immediately; render() replaces them with the resolved
// neighborhood + city name once the grade fetch returns.
locLabel.textContent = `${lat.toFixed(3)}, ${lon.toFixed(3)}`;
@ -58,6 +69,7 @@ const hero = document.getElementById("hero");
const heroLocate = document.getElementById("hero-locate");
const heroPick = document.getElementById("hero-pick");
const heroMsg = document.getElementById("hero-locate-msg");
const geoBox = document.getElementById("geo-approx");
function locateMsg(text) {
if (!heroMsg) return;
@ -106,6 +118,16 @@ heroLocate?.addEventListener("click", () => {
? "That took too long. Try again, or pick a spot on the map."
: "Your location isn't available right now. Pick a spot on the map instead.",
);
// The dead end this fallback exists for. If the server can offer a
// coarse guess, take over the message — telling someone to "pick a spot
// on the map" while a guessed place is already loading reads as two
// contradictory instructions.
tryApprox().then((ok) => {
if (!ok) return;
locateMsg(err.code === err.PERMISSION_DENIED
? "No problem — location permission stays off."
: "Your device's location wasn't available.");
});
},
{ enableHighAccuracy: false, timeout: 10000, maximumAge: 600000 },
);
@ -113,6 +135,30 @@ heroLocate?.addEventListener("click", () => {
heroPick?.addEventListener("click", () => { locateMsg(""); findBtn.click(); });
// ---- approximate-location fallback ----
// Only ever a rescue for someone who has no location at all: it runs after a
// declined/failed geolocation prompt, or on load in a context where the browser
// will not offer geolocation at all (plain HTTP). It never runs for a visitor
// who already has a place — theirs is better than any guess — and never
// pre-empts the permission prompt.
let approxTried = false;
async function tryApprox() {
if (approxTried || selected) return false;
approxTried = true;
const data = await fetchApprox();
// 204/anything unusable → exactly today's behaviour, nothing rendered.
if (!data || selected) return false;
renderApprox(geoBox, data, () => { locateMsg(""); findBtn.click(); });
selectLocation(data.lat, data.lon, { approx: true, label: data.label });
return true;
}
// No geolocation available at all (insecure context, or a browser without it)
// and nothing remembered: there is no prompt to decline, so offer the guess up
// front rather than leaving the page with no place at all.
if (!canLocate && !loadLastLocation()) tryApprox();
/** Re-point the hero card at the place the visitor actually chose. */
function updateHero(data) {
if (!hero) return;
@ -121,7 +167,12 @@ function updateHero(data) {
const g = targetDay && (targetDay.tmax || targetDay.tmin);
if (!card || !g) return;
document.getElementById("hero-where").textContent = `Today in ${placeLabel(data)}`;
// A guessed place is reverse-geocoded like any other cell, which would name a
// *neighbourhood* — far more precision than an IP lookup ever had. Say
// "roughly near <city>" instead, so the headline can't overstate what we know.
document.getElementById("hero-where").textContent =
approx ? `Roughly near ${approxLabel || placeLabel(data)}`
: `Today in ${placeLabel(data)}`;
document.getElementById("hero-band").textContent = g.grade;
document.getElementById("hero-detail").textContent =
`${targetDay.tmax === g ? "High temp" : "Low temp"} in the ${pctOrd(g.percentile)} percentile`;
@ -267,7 +318,8 @@ function render(data) {
results.innerHTML = `
<div class="loc-head">
<div class="loc-title">
<h2>${placeLabel(data)}</h2>
<h2>${approx ? `Near ${esc(approxLabel || placeLabel(data))}` : placeLabel(data)}</h2>
${approx ? `<p class="loc-approx-tag">approximate — from your connection, not your device</p>` : ""}
<ul class="loc-meta">
${data.place ? `<li><b>${coords}</b></li>` : ""}
<li>Climatology from <b>${yrs}</b></li>
@ -605,6 +657,10 @@ function flashBtn(id, text) {
function updateHash() {
if (!selected) return;
// A guessed location is deliberately not remembered and not written into the
// URL: next visit must ask again rather than treating a guess as a choice,
// and a shared link must never carry a rough location the sender never picked.
if (approx) return;
history.replaceState(null, "", locHash(selected.lat, selected.lon, dateInput.value));
saveLastLocation(selected.lat, selected.lon, dateInput.value);
}

84
frontend/static/geoip.js Normal file
View file

@ -0,0 +1,84 @@
// Approximate-location fallback.
//
// The only reason this exists: a visitor who declines (or cannot use) browser
// geolocation currently hits a dead end — search or the map picker, or nothing.
// This asks the backend for a coarse guess derived from the request's own IP,
// and offers it as a *suggestion*.
//
// Three rules the UI must never break, because the accuracy gap is enormous
// (browser geolocation is metres; a city-level IP database is tens of
// kilometres, against a ~2 mile grid cell):
//
// 1. Say it is a guess, in the visitor's own words, right next to the result.
// Never present it as "your location".
// 2. Put the correction one obvious tap away, permanently visible — not
// hidden behind a dismissed toast.
// 3. Never persist it. An explicit choice is remembered (localStorage) and
// written into the URL hash; a guess is neither. A guess that became
// sticky would be indistinguishable from a choice on the next visit, and
// a guessed URL would propagate someone else's rough location when shared.
//
// The endpoint answers 204 for every "no" — feature off, no database, private
// or unparsable address, no match, match too coarse — so there is exactly one
// branch here and it is "behave exactly as the site does today".
import { uv } from "./account.js";
import { track } from "./digest.js";
import { esc } from "./shared.js";
/**
* Ask the backend for an approximate location.
* Resolves to the suggestion object, or null when there is nothing to suggest.
* Never rejects: a failed fetch is just "no suggestion".
*/
export async function fetchApprox() {
try {
const res = await fetch(uv("geoip"), { headers: { Accept: "application/json" } });
if (res.status !== 200) return null; // 204 is the normal "no"
const data = await res.json();
if (!data || !data.approximate) return null; // never trust an unlabelled payload
if (typeof data.lat !== "number" || typeof data.lon !== "number") return null;
return data;
} catch (e) {
return null;
}
}
function attributionHtml(a) {
// Both candidate databases are free *with attribution* and both ask for the
// credit on the page that displays a result — so it rides with the result
// rather than living only in a footer that this view doesn't have.
if (!a || !a.text) return "";
const text = esc(a.text);
return a.url
? ` <a class="geo-approx-credit" href="${esc(a.url)}" rel="noopener nofollow" target="_blank">${text}</a>`
: ` <span class="geo-approx-credit">${text}</span>`;
}
/**
* Render the suggestion banner into `host` and load the guessed place.
*
* @param host container element (hidden until there is something to show)
* @param data a suggestion from fetchApprox()
* @param onFix called when the visitor corrects the guess
*/
export function renderApprox(host, data, onFix) {
if (!host) return;
host.innerHTML =
`<p class="geo-approx-line">Showing weather near <b>${esc(data.label)}</b>.</p>` +
`<p class="geo-approx-note">That is a rough guess from your internet connection, ` +
`not from your device — it can be tens of kilometres out.` +
attributionHtml(data.attribution) + `</p>` +
`<button type="button" class="btn-ghost geo-approx-fix">Not right? Choose your spot</button>`;
host.hidden = false;
host.querySelector(".geo-approx-fix")?.addEventListener("click", () => {
track("home.geoip_fixed");
onFix?.();
});
track("home.geoip_shown");
}
export function clearApprox(host) {
if (!host) return;
host.hidden = true;
host.innerHTML = "";
}

View file

@ -1962,6 +1962,24 @@ details.hub-country > summary:hover { color: var(--accent); }
.hero-jump a { color: var(--muted); text-decoration: none; }
.hero-jump a:hover { color: var(--accent); }
/* Approximate-location banner (static/geoip.js). Read as an aside, not a
result: recessed surface, no accent fill, and the correction button is a
full-width ~44px target so it is never the hard thing to find on a phone. */
.geo-approx {
display: flex; flex-direction: column; gap: 8px;
margin: 0; padding: 12px 14px; max-width: 42ch;
background: var(--surface-2); border: 1px dashed var(--border); border-radius: 10px;
}
.geo-approx[hidden] { display: none; }
.geo-approx-line { margin: 0; font-size: 14px; }
.geo-approx-note { margin: 0; font-size: 12px; color: var(--muted); }
.geo-approx-credit { color: var(--muted); text-decoration: underline; }
.geo-approx-credit:hover { color: var(--accent); }
.geo-approx .geo-approx-fix { min-height: 44px; font-size: 16px; width: 100%; }
/* The same "this is a guess" claim on the results heading, so it survives a
scroll past the hero. */
.loc-approx-tag { margin: 2px 0 0; font-size: 12px; color: var(--muted); }
/* Desktop: copy beside the card rather than stacked above it. */
@media (min-width: 900px) {
.hero { grid-template-columns: 1.1fr 1fr; align-items: center; gap: 40px; }

View file

@ -68,6 +68,13 @@
<button type="button" id="hero-pick" class="btn-ghost">Pick on the map</button>
</div>
<p class="hero-locate-msg" id="hero-locate-msg" role="status" aria-live="polite" hidden></p>
{# Approximate-location fallback (static/geoip.js). Server-rendered EMPTY
and hidden on purpose: this page's HTML must stay byte-identical for
every visitor so its ETag keeps validating and no shared cache can
ever hand one person's city to another. The suggestion is fetched
client-side, and only for a visitor who ended up with no location at
all. #}
<div class="geo-approx" id="geo-approx" role="status" aria-live="polite" hidden></div>
<p class="hero-jump"><a href="#panel">See your week ↓</a></p>
</div>
</section>

View file

@ -12,9 +12,27 @@
there is very little about you to collect in the first place.</p>
<h2>Your location</h2>
{# This section states a fact about what the server actually does, so it follows
the THERMOGRAPH_GEOIP flag instead of being written once and left to rot.
See content.py's geoip_enabled(). #}
{% if geoip %}
<p>The precise way the site learns where you are is if you press <b>Use my location</b>,
which asks your browser for permission. You can decline, and picking a place on the map
works just as well.</p>
<p>If you decline, or your browser cannot offer it, we fall back to a rough guess at your
city from your IP address so there is something local to show instead of nothing. It is
always labelled as a guess, never as your position — it is accurate to tens of kilometres
at best, and one tap corrects it. It is also not remembered: a guess is never saved in
your browser or written into the page address, so only a place you actually pick is kept.</p>
<p>That lookup runs entirely on our own server, against a database file we host ourselves.
Your IP address is never sent to another company, and it is not stored: it is read from
the request, used to find a city, and discarded. It is deliberately kept out of our
request log for this feature too, so nothing anywhere links an address to a place.</p>
{% else %}
<p>Thermograph never looks up your location from your IP address. The only way the site
learns where you are is if you press <b>Use my location</b>, which asks your browser for
permission. You can decline, and picking a place on the map works just as well.</p>
{% endif %}
<p>Whichever way you choose a place, the coordinates are sent to the server only as part of
the ordinary request for that grid square's climate data, and are not logged beyond the
normal web-server request handling every site does. The place you picked is remembered in

View file

@ -0,0 +1,58 @@
"""Approximate-location fallback, frontend side.
Two things are worth locking down here, and both are about honesty rather than
mechanics: the server-rendered HTML must not vary by visitor (so no shared
cache can leak one person's city to another), and the privacy page's factual
claim about IP lookups must track the flag that decides whether they happen.
"""
import content
B = "/thermograph"
def test_home_ships_an_empty_hidden_container(client):
"""The suggestion is fetched client-side. If it were ever server-rendered
into the homepage, that page's HTML — and its ETag — would vary per
visitor, which is exactly the cache-poisoning shape this design avoids."""
r = client.get(f"{B}/")
assert r.status_code == 200
assert 'id="geo-approx"' in r.text
assert "hidden" in r.text.split('id="geo-approx"')[1].split(">")[0]
def test_home_html_is_identical_regardless_of_client_address(client):
"""Same bytes, same ETag, whatever the request claims about where it came
from the property that lets the homepage stay cacheable."""
a = client.get(f"{B}/", headers={"X-Forwarded-For": "8.8.8.8"})
b = client.get(f"{B}/", headers={"X-Forwarded-For": "1.1.1.1"})
assert a.text == b.text
assert a.headers["etag"] == b.headers["etag"]
def test_privacy_says_no_ip_lookup_while_the_flag_is_off(client, monkeypatch):
monkeypatch.delenv("THERMOGRAPH_GEOIP", raising=False)
r = client.get(f"{B}/privacy")
assert r.status_code == 200
assert "never looks up your location from your IP address" in r.text
def test_privacy_describes_the_fallback_when_the_flag_is_on(client, monkeypatch):
"""Turning the feature on must not leave a published claim behind that is
no longer true."""
monkeypatch.setenv("THERMOGRAPH_GEOIP", "1")
r = client.get(f"{B}/privacy")
assert r.status_code == 200
assert "never looks up your location from your IP address" not in r.text
assert "rough guess" in r.text
assert "not stored" in r.text
def test_geoip_enabled_reads_the_shared_flag(monkeypatch):
monkeypatch.delenv("THERMOGRAPH_GEOIP", raising=False)
assert content.geoip_enabled() is False
for on in ("1", "true", "YES", "on"):
monkeypatch.setenv("THERMOGRAPH_GEOIP", on)
assert content.geoip_enabled() is True, on
for off in ("0", "", "no", "maybe"):
monkeypatch.setenv("THERMOGRAPH_GEOIP", off)
assert content.geoip_enabled() is False, off

View file

@ -0,0 +1,18 @@
[Unit]
Description=Refresh the Thermograph IP-geolocation database
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
# GEOIP_URL / GEOIP_BASE / THERMOGRAPH_GEOIP_HOST_DIR can be overridden here if
# the deployment switches databases. The leading `-` keeps a missing file
# non-fatal, matching thermograph.service.
EnvironmentFile=-/etc/thermograph.env
ExecStart=/opt/thermograph/infra/deploy/geoip-refresh.sh
# Downloading ~125 MB and verifying it needs no privilege beyond writing the
# destination directory, and nothing else on the box.
ProtectSystem=strict
ReadWritePaths=/var/lib/thermograph/geoip
PrivateTmp=yes
NoNewPrivileges=yes

90
infra/deploy/geoip-refresh.sh Executable file
View file

@ -0,0 +1,90 @@
#!/usr/bin/env bash
# Refresh the IP-geolocation database used by the approximate-location fallback
# (backend/data/geoip.py). Runs on the HOST (prod/beta), not in a container, on
# a systemd timer — see geoip-refresh.timer.
#
# Why on the host and not baked into the image:
# * it is a 100-200 MB data file on its own monthly release cadence, not code;
# baking it in adds that to every image pull and freezes its freshness to
# whenever the last deploy happened;
# * MaxMind's GeoLite EULA *requires* deleting a database within 30 days of a
# newer release, which a build-time copy cannot honour on its own;
# * geoip.py memory-maps the file and re-opens it when (mtime, size) changes,
# so an atomic swap here is picked up with no restart and no deploy.
#
# Default source: DB-IP's IP to City Lite, CC BY 4.0, no account and no licence
# key — which is why this script needs no secret from the vault. Point
# GEOIP_URL elsewhere (e.g. a MaxMind permalink carrying a licence key from
# /etc/thermograph.env) to use a different database; geoip.py reads the file's
# own metadata for the attribution line, so nothing else changes.
#
# sudo /opt/thermograph/infra/deploy/geoip-refresh.sh
#
# Exit status: 0 on a successful swap AND on "already current" (so the timer
# doesn't alarm); non-zero only on a real failure, leaving any existing
# database untouched — a failed refresh must never be worse than a stale one.
set -euo pipefail
DEST_DIR="${THERMOGRAPH_GEOIP_HOST_DIR:-/var/lib/thermograph/geoip}"
DEST="$DEST_DIR/city.mmdb"
# DB-IP publishes one file per month at a predictable URL. Try this month, then
# last month: the new file appears at some point on the 1st, so a timer firing
# early would otherwise fail for a day.
BASE="${GEOIP_BASE:-https://download.db-ip.com/free}"
THIS_MONTH="$(date -u +%Y-%m)"
LAST_MONTH="$(date -u -d "$(date -u +%Y-%m-01) -1 month" +%Y-%m)"
log() { echo "[geoip-refresh] $*"; }
mkdir -p "$DEST_DIR"
tmp="$(mktemp -d "${TMPDIR:-/tmp}/geoip.XXXXXX")"
trap 'rm -rf "$tmp"' EXIT
url=""
if [ -n "${GEOIP_URL:-}" ]; then
url="$GEOIP_URL"
else
for m in "$THIS_MONTH" "$LAST_MONTH"; do
candidate="$BASE/dbip-city-lite-$m.mmdb.gz"
if curl -fsSI --max-time 30 "$candidate" >/dev/null 2>&1; then
url="$candidate"; break
fi
done
fi
[ -n "$url" ] || { log "no published database found at $BASE for $THIS_MONTH or $LAST_MONTH"; exit 1; }
log "source: $url"
curl -fsSL --max-time 900 --retry 3 --retry-delay 10 -o "$tmp/db.gz" "$url"
gunzip -c "$tmp/db.gz" > "$tmp/city.mmdb"
# Prove the file actually opens and answers before it can replace a working
# one. A truncated download that merely *exists* would otherwise silently
# disable lookups for a month.
python3 - "$tmp/city.mmdb" <<'PY'
import sys
import maxminddb
with maxminddb.open_database(sys.argv[1]) as r:
meta = r.metadata()
if "city" not in (meta.database_type or "").lower():
sys.exit(f"unexpected database_type: {meta.database_type}")
if r.get("8.8.8.8") is None and r.get("1.1.1.1") is None:
sys.exit("database opened but answered nothing for known public addresses")
print(f"[geoip-refresh] verified {meta.database_type} "
f"built {meta.build_epoch} nodes={meta.node_count}")
PY
if [ -f "$DEST" ] && cmp -s "$tmp/city.mmdb" "$DEST"; then
log "already current: $DEST unchanged"
exit 0
fi
# Atomic swap on the same filesystem: readers either see the whole old file or
# the whole new one, never a half-written mmap. The app's open handle keeps the
# unlinked inode alive until it re-opens on the mtime change.
install -m 0444 "$tmp/city.mmdb" "$DEST.new"
mv -f "$DEST.new" "$DEST"
log "installed $DEST ($(stat -c %s "$DEST") bytes)"
# MaxMind's EULA requires deleting superseded copies within 30 days; keeping no
# copies at all satisfies that trivially and is the right default for DB-IP too.
find "$DEST_DIR" -maxdepth 1 -name 'city.mmdb.*' -type f -delete 2>/dev/null || true

View file

@ -0,0 +1,18 @@
[Unit]
Description=Monthly refresh of the Thermograph IP-geolocation database
[Timer]
# Both candidate databases publish monthly. Weekly rather than monthly on
# purpose: the script exits 0 without touching anything when the file is already
# current, so the extra runs cost one HEAD request, and they mean a missed or
# failed monthly run self-heals within a week instead of leaving the data a
# month stale. (MaxMind additionally *requires* deleting a superseded copy
# within 30 days, which a strictly-monthly timer would sit right on the edge of.)
OnCalendar=Mon 04:17 UTC
# Don't have every host in the fleet hit the mirror at the same second.
RandomizedDelaySec=2h
# Catch up after a reboot that spanned the scheduled time.
Persistent=true
[Install]
WantedBy=timers.target

View file

@ -207,3 +207,37 @@ THERMOGRAPH_BASE_URL=https://thermograph.org
# required — Discord delivers content for mentions/DMs. Unset/0 => no gateway
# connection. (Code: thermograph-backend notifications/discord_bot.py.)
#THERMOGRAPH_DISCORD_BOT=1
# --- approximate-location fallback (backend/data/geoip.py) ------------------
# When a visitor declines (or cannot use) browser geolocation, suggest a coarse
# city from their IP so the site has something local to show. OFF by default,
# and inert until deploy/geoip-refresh.sh has landed a database at
# /var/lib/thermograph/geoip/city.mmdb — with either missing, the site behaves
# exactly as it does today (the endpoint answers 204 and the UI does nothing).
#
# Read by BOTH containers: the backend does the lookup; the frontend reads it
# only to decide which paragraph /privacy renders about location. Turning this
# on therefore changes the published privacy statement in the same deploy, on
# purpose — do not set it in one place and not the other.
#
# The lookup is local (a file on our own disk, MaxMind MMDB format) — the
# visitor's IP never leaves the server, so no third party becomes a data
# processor. The IP is read in memory and discarded: it is written to no
# database, no metric, and not even the access log (that route is excluded).
#THERMOGRAPH_GEOIP=1
# Where the refresh job writes the database on the host; bind-mounted read-only
# into the backend container at /geoip. Only override if the default path is
# unsuitable.
#THERMOGRAPH_GEOIP_HOST_DIR=/var/lib/thermograph/geoip
# Reject a record the database itself says is worse than this many km — that is
# what stops a country-centroid guess being presented as a city. Default 100.
#THERMOGRAPH_GEOIP_MAX_RADIUS_KM=100
# Credit line shown beside any suggestion, as "text|url". Normally unset: it is
# derived from the installed database's own metadata, so swapping DB-IP for
# GeoLite2 changes the attribution automatically. Set it only for a database
# whose metadata we don't recognise.
#THERMOGRAPH_GEOIP_ATTRIBUTION=IP Geolocation by DB-IP|https://db-ip.com
# Alternate download source for deploy/geoip-refresh.sh. Unset = DB-IP IP to
# City Lite (CC BY 4.0, no account, no licence key — which is why nothing here
# is a secret). A MaxMind permalink would embed a licence key and therefore
# belongs in the SOPS vault rather than in this example file.
#GEOIP_URL=

View file

@ -129,6 +129,13 @@ services:
# 503 and climate.py falls straight through to NASA — an accelerator,
# never a dependency.
THERMOGRAPH_LAKE_URL: http://lake:8141
# Approximate-location fallback (backend/data/geoip.py). OFF unless
# THERMOGRAPH_GEOIP is set truthy in /etc/thermograph.env, and inert even
# then until deploy/geoip-refresh.sh has landed a database at the host
# path bind-mounted below — so shipping the code changes nothing by
# itself, and the mount being empty is a supported steady state.
THERMOGRAPH_GEOIP: ${THERMOGRAPH_GEOIP:-}
THERMOGRAPH_GEOIP_DB: /geoip/city.mmdb
# Prod secrets live in /etc/thermograph.env: POSTGRES_PASSWORD,
# THERMOGRAPH_AUTH_SECRET, THERMOGRAPH_VAPID_PRIVATE_KEY/_PUBLIC_KEY,
# THERMOGRAPH_COOKIE_SECURE=1, mail/Discord keys, ... (see
@ -145,6 +152,12 @@ services:
# (above) points runtime state here instead, clear of the code.
- appdata:/state
- applogs:/app/logs
# The IP-geolocation database, refreshed on the HOST by
# deploy/geoip-refresh.sh (systemd timer, monthly) and mounted read-only:
# the app must never be able to write it, and a 100-200 MB data file that
# goes stale on its own schedule does not belong baked into the image.
# geoip.py re-opens on mtime change, so a refresh needs no restart.
- ${THERMOGRAPH_GEOIP_HOST_DIR:-/var/lib/thermograph/geoip}:/geoip:ro
# No compose-level healthcheck override -- the image's own Dockerfile
# HEALTHCHECK (port-aware via ${PORT}) already covers this, and frontend's
# depends_on below reads it the same way.
@ -208,6 +221,13 @@ services:
PORT: 8080
THERMOGRAPH_SERVICE_ROLE: frontend
THERMOGRAPH_API_BASE_INTERNAL: http://backend:8137
# Read for ONE purpose: which paragraph the /privacy page renders about
# location. Frontend behaviour does not branch on it — the interactive
# tool simply calls the backend endpoint and gets a 204 when the feature
# is off. Same value as backend's, from the same /etc/thermograph.env, so
# the published privacy statement can never drift from what the server
# actually does.
THERMOGRAPH_GEOIP: ${THERMOGRAPH_GEOIP:-}
# No volumes -- stateless, holds no data of its own.
cpus: ${FRONTEND_CPUS:-2}
deploy: