Approximate-location fallback from the client IP (feature-flagged off)
All checks were successful
PR build (required check) / changes (pull_request) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Successful in 1m8s
PR build (required check) / build-backend (pull_request) Successful in 1m36s
PR build (required check) / gate (pull_request) Successful in 2s
All checks were successful
PR build (required check) / changes (pull_request) Successful in 6s
secrets-guard / encrypted (pull_request) Successful in 5s
PR build (required check) / validate-observability (pull_request) Has been skipped
PR build (required check) / build-frontend (pull_request) Successful in 1m8s
PR build (required check) / build-backend (pull_request) Successful in 1m36s
PR build (required check) / gate (pull_request) Successful in 2s
A visitor who declines browser geolocation currently has no location at all — the copy sends them to the map picker. This adds an opt-in fallback that suggests a coarse city from the request's own IP, presented as a guess with a one-tap correction, so the dead end has a way out. backend/data/geoip.py does the lookup against a local MMDB file (DB-IP IP to City Lite or GeoLite2 City — same format, either works, attribution derived from the file's metadata). Off unless THERMOGRAPH_GEOIP is truthy AND the database exists AND maxminddb imports; every degraded case — private/reserved/ CGNAT/IPv6-ULA addresses, unparseable input, no record, a country-centroid record with no city, a record wider than the accuracy limit, a corrupt file — returns None, which the route answers as 204 and the client treats exactly as today. The IP is read in memory and dropped. GET /api/v2/geoip runs no RunAudit, records no metric dimension, and is excluded from the access log (its own "geoip" traffic category) so no stored, joinable (IP, location) pair is ever written. The response is no-store/Vary:* and takes no parameters. Client-side rather than server-rendered on purpose: the homepage's HTML and weak ETag must stay byte-identical for every visitor, or any cache added in front of it can serve one visitor's city to another. The suggestion is fetched only after a declined/unavailable prompt, and only when nothing is remembered. A guess is never persisted — no localStorage write, no URL hash — and the hero and results headings say "roughly near"/"near … approximate" rather than letting the reverse-geocoded cell name a neighbourhood the lookup never knew. The /privacy page's "never looks up your location from your IP address" paragraph now follows the same flag out of the same env file, so the published statement and the behaviour flip together. Database lifecycle is a host-side systemd timer (infra/deploy/geoip-refresh.*) that verifies a download opens and answers before swapping it in atomically, bind-mounted read-only; geoip.py re-opens on mtime change, so a refresh needs no restart. GEOIP-APPROX-LOCATION.md carries the database comparison, the SSR-vs-client argument, the privacy analysis, and the open decisions.
This commit is contained in:
parent
16bddb6625
commit
083d3bd0f8
19 changed files with 1694 additions and 4 deletions
489
GEOIP-APPROX-LOCATION.md
Normal file
489
GEOIP-APPROX-LOCATION.md
Normal file
|
|
@ -0,0 +1,489 @@
|
|||
# Approximate-location fallback — design + prototype
|
||||
|
||||
Status: **prototype on `feat/geoip-approx-location`, feature-flagged OFF, not
|
||||
deployed and not merged.** Nothing in this branch changes production behaviour
|
||||
until `THERMOGRAPH_GEOIP` is set *and* a database file exists on the host.
|
||||
|
||||
This doc is the decision record. If it survives review it should move to
|
||||
`thermograph-docs` (per the root `CLAUDE.md`: cross-cutting decision docs live
|
||||
there); it sits here for now so the reasoning ships with the code it describes.
|
||||
|
||||
---
|
||||
|
||||
## 1. The problem
|
||||
|
||||
Location today comes from exactly one source: `navigator.geolocation.getCurrentPosition`
|
||||
in `frontend/static/app.js`, gated on `window.isSecureContext`. If the visitor
|
||||
declines the permission prompt, the prompt times out, or the browser won't offer
|
||||
geolocation at all, the site has **no location** and the copy says "Pick a spot
|
||||
on the map instead."
|
||||
|
||||
That is a dead end for the exact visitor the homepage is written for — someone
|
||||
who arrived to answer "how unusual is *my* weather?" and has just been handed a
|
||||
world map. This change gives that person something local, honestly labelled.
|
||||
|
||||
## 2. What was built
|
||||
|
||||
| Piece | File |
|
||||
| --- | --- |
|
||||
| Lookup module (flag, parsing, rejection rules, attribution) | `backend/data/geoip.py` |
|
||||
| Endpoint `GET /api/v2/geoip` | `backend/web/app.py` (`api_geoip`) |
|
||||
| Access-log exclusion + traffic category | `backend/web/app.py`, `backend/core/metrics.py` |
|
||||
| Event names `home.geoip_shown` / `home.geoip_fixed` | `backend/core/metrics.py` |
|
||||
| Client fetch + suggestion banner | `frontend/static/geoip.js` |
|
||||
| Wiring, non-persistence, honest headings | `frontend/static/app.js` |
|
||||
| Banner container (empty, hidden) | `frontend/templates/home.html.j2` |
|
||||
| Banner styles | `frontend/static/style.css` |
|
||||
| Flag-aware privacy copy | `frontend/content.py`, `frontend/templates/privacy.html.j2` |
|
||||
| Host refresh job + timer | `infra/deploy/geoip-refresh.{sh,service,timer}` |
|
||||
| Read-only DB mount + env plumbing | `infra/docker-compose.yml`, `infra/deploy/thermograph.env.example` |
|
||||
| Tests | `backend/tests/data/test_geoip.py`, `backend/tests/web/test_geoip_api.py`, `frontend/tests/unit/test_geoip_ui.py` |
|
||||
|
||||
`cd backend && make test` → **449 passed, 8 skipped** (was 386/7).
|
||||
`cd frontend && make test-unit` → **41 passed**.
|
||||
|
||||
---
|
||||
|
||||
## 3. Database choice — recommendation: **DB-IP IP to City Lite**
|
||||
|
||||
| | **DB-IP IP to City Lite** | MaxMind GeoLite2 City | IP2Location LITE DB5/DB11 |
|
||||
| --- | --- | --- | --- |
|
||||
| Licence | **CC BY 4.0** | GeoLite End User License Agreement (incorporates CC BY-SA 4.0 terms) | CC BY-SA 4.0 / "free with attribution" |
|
||||
| Account needed | **No** — direct download URL | **Yes** — account + licence key | **Yes** — free account |
|
||||
| Secret to manage | **None** | Licence key (→ SOPS vault, → CI, → rotation) | Account credentials |
|
||||
| Attribution | `<a href="https://db-ip.com">IP Geolocation by DB-IP</a>` on pages showing results | "This product includes GeoLite Data created by MaxMind, available from https://www.maxmind.com" | "uses the IP2Location LITE database for IP geolocation", linked |
|
||||
| Retention obligation | none | **must delete a database within 30 days of a newer release**; 30 downloads/day cap | none |
|
||||
| Update cadence | monthly | weekly (Tue/Fri) | monthly |
|
||||
| Size (current release) | 125 MB MMDB (685 MB CSV), ~8.05 M records | ~60–70 MB MMDB | 69 MB BIN (IPv4) + 171 MB BIN (IPv6), separate files |
|
||||
| Format / reader | **MMDB → `maxminddb`** (pure-Python, mmap) | MMDB → `maxminddb` / `geoip2` | proprietary BIN → vendor `IP2Location` package (or CSV you index yourself) |
|
||||
| Accuracy fields | city + lat/lon, **no `accuracy_radius`** | city + lat/lon + `accuracy_radius` | city + lat/lon |
|
||||
|
||||
**Recommend DB-IP IP to City Lite.** Reasoning, in priority order:
|
||||
|
||||
1. **No credential.** MaxMind means a licence key in the SOPS vault, in the
|
||||
deploy path, and in a rotation story — new secret surface for a feature whose
|
||||
entire justification is privacy hygiene. DB-IP needs none, so
|
||||
`geoip-refresh.sh` reads nothing from the vault at all.
|
||||
2. **No retention obligation.** GeoLite's EULA requires deleting a database
|
||||
within 30 days of a newer release. That is a *compliance* clock on an ops
|
||||
job, and a missed job becomes a licence breach rather than merely stale data.
|
||||
CC BY 4.0 has no such clause.
|
||||
3. **Same format, same reader.** Both are MMDB, so this is a swap of one file
|
||||
and one URL, not a code change — `geoip.py` reads the raw record dict and
|
||||
derives its attribution line from the file's own metadata. If accuracy turns
|
||||
out to be materially worse in practice, moving to GeoLite2 is a config change
|
||||
plus a vault entry, and the code is already written to accept it.
|
||||
4. **Attribution is a link, not a legal paragraph.** One muted anchor in the
|
||||
suggestion banner satisfies CC BY 4.0 and DB-IP's stated wording.
|
||||
|
||||
**Against IP2Location LITE**: a proprietary BIN format needing the vendor's own
|
||||
Python package, IPv4 and IPv6 as *separate* databases (two files, two readers,
|
||||
two refresh paths), an account to download, and CC BY-**SA** — a share-alike
|
||||
term that is noise to reason about for a file we merely read.
|
||||
|
||||
**Cost of the DB-IP choice, stated plainly:** the Lite file carries no
|
||||
`accuracy_radius`, so the "reject anything the database admits is too coarse"
|
||||
guard degrades to "reject anything with no city name". That is still the guard
|
||||
that matters (a country centroid is the damaging case), but it is weaker.
|
||||
`suggestion_from_record()` applies the radius rule whenever the field *is*
|
||||
present, so switching to GeoLite2 turns the stronger guard back on for free.
|
||||
|
||||
**Reader library:** `maxminddb` (pinned `2.8.2` in `backend/requirements.txt`),
|
||||
not `geoip2`. `geoip2` adds a model layer over the same reader and type-checks
|
||||
`database_type` against MaxMind's own names; we read four fields out of a plain
|
||||
dict, and staying on `maxminddb` keeps DB-IP and MaxMind files equally usable.
|
||||
It is imported *lazily inside* `geoip.py`, so a missing wheel disables the
|
||||
feature rather than breaking boot.
|
||||
|
||||
---
|
||||
|
||||
## 4. Where it hooks in — recommendation: **client-side, on demand**
|
||||
|
||||
Two options were on the table.
|
||||
|
||||
**A. Server-side during SSR of `/`.** `frontend/content.py`'s `home_page()`
|
||||
would look up the IP and render the guess into the first paint.
|
||||
|
||||
**B. Client-side after geolocation is declined/unavailable.** The browser calls
|
||||
`GET /api/v2/geoip` and swaps the banner in. **This is what the prototype does.**
|
||||
|
||||
### Why B
|
||||
|
||||
**Caching is the deciding factor, and the current setup is more fragile than it
|
||||
looks.** `frontend/content.py`'s `_respond_html()` returns HTML with a weak
|
||||
ETag (`W/"<sha1 of the rendered body>"`) and **no `Cache-Control` and no
|
||||
`Vary`**. The host Caddy (`infra/deploy/Caddyfile`) only does `encode` +
|
||||
`reverse_proxy` — no cache module today. So right now nothing caches the
|
||||
homepage, but nothing *declares* that it mustn't either.
|
||||
|
||||
Fold a location into that response and three things break at once:
|
||||
|
||||
- The ETag becomes a function of the visitor's city. Two visitors flap each
|
||||
other's validators; conditional requests stop being useful, and a returning
|
||||
visitor whose IP maps elsewhere gets a spurious 200 instead of a 304.
|
||||
- Any cache added later — a Caddy `cache` directive, Cloudflare in front of the
|
||||
apex, a corporate proxy on the visitor's side — will serve one visitor's city
|
||||
to another, because the response carries nothing saying it varies. There is no
|
||||
correct `Vary` header for this: it varies on `X-Forwarded-For`, which is not a
|
||||
header a cache keys on, and `Vary: *` would make the homepage permanently
|
||||
uncacheable for everyone to serve a fallback that only a minority need.
|
||||
- The SSR path would also have to carry the client IP over the extra
|
||||
frontend→backend hop (`api_client.py` currently forwards only `Host` and
|
||||
`X-Forwarded-Proto`). That means propagating the IP into a second process's
|
||||
request path — more places it can end up in a log, for no benefit.
|
||||
|
||||
**The first-paint argument is weaker here than it looks**, because the homepage
|
||||
hero is *already* server-rendered with real content: the most unusual city
|
||||
we're currently tracking, with its grade, band colour and percentile. There is
|
||||
no blank space for the guess to fill. The suggestion swaps into a page that is
|
||||
already complete, and it costs one small JSON request with no data dependency in
|
||||
front of it.
|
||||
|
||||
**And SSR would do the lookup for the wrong people.** The server cannot see
|
||||
`localStorage`, so it cannot know that most returning visitors already have a
|
||||
saved place that is strictly better than any guess. An SSR implementation would
|
||||
geolocate every homepage request, including the majority who will never use the
|
||||
result — the opposite of "touch the IP as little as possible". Option B only
|
||||
runs after the visitor has demonstrably ended up with nothing.
|
||||
|
||||
### What B costs
|
||||
|
||||
- **Needs JavaScript.** A no-JS visitor sees today's behaviour. Acceptable: the
|
||||
entire interactive tool already needs JS.
|
||||
- **One extra round trip** after the decline, ~200 bytes each way.
|
||||
- **A 204 for everyone in the fallback path when the feature is off.** The
|
||||
alternative — giving the frontend its own copy of the flag to decide whether
|
||||
to fetch — is two flags to keep in sync and a new way for the UI and the
|
||||
server to disagree. One flag, one 204, is the better trade.
|
||||
|
||||
### If you overrule this and want SSR anyway
|
||||
|
||||
Do it as a *separate* route (`/near` or `?near=1`), not on the cached `/`, and
|
||||
give that route `Cache-Control: private, no-store`. Never make the canonical,
|
||||
indexable homepage location-varying — that is also an SEO hazard: a crawler in
|
||||
one datacentre would index a page claiming a city, and different crawler IPs
|
||||
would see different "canonical" content.
|
||||
|
||||
---
|
||||
|
||||
## 5. Privacy design
|
||||
|
||||
### The IP is never persisted — mechanically, not by convention
|
||||
|
||||
- `geoip.lookup(ip)` takes a string, memory-maps a file, returns a dict. It
|
||||
writes nothing, logs nothing, and never raises.
|
||||
- `api_geoip` uses **no `RunAudit`** (which would record the derived lat/lon)
|
||||
and adds **no metric dimension** (the counters are aggregate by category).
|
||||
- The route is excluded from the access log. `metrics.classify_inbound()` now
|
||||
returns a dedicated `"geoip"` category and the middleware in `web/app.py`
|
||||
skips `audit.log_access` for it. Every other route still logs, and
|
||||
`test_a_normal_route_still_is_logged` guards that exclusion from widening.
|
||||
|
||||
This is the important one. Today the access log stores a raw IP per request.
|
||||
Without the exclusion, the *one* endpoint whose whole job is turning an IP
|
||||
into a place would leave a stored, timestamped, joinable (IP,
|
||||
approximate-location) pair on disk and in Loki. That artefact — not the
|
||||
transient lookup — is what would actually constitute keeping location data
|
||||
about identifiable people. The exclusion is not defence in depth; it is the
|
||||
substance of the claim.
|
||||
|
||||
This does **not** depend on the separate access-log IP-truncation work. That
|
||||
change is still worth doing, and it composes: truncation reduces what the
|
||||
*other* routes retain, while this exclusion means the geoip route retains
|
||||
nothing at all either way.
|
||||
|
||||
- `Cache-Control: no-store, private` + `Vary: *` on the response, so it cannot
|
||||
be stored or reused by any intermediary.
|
||||
- The endpoint takes **no parameters**. You cannot ask it about somebody else's
|
||||
address; the only input is the connection you already own. `?ip=` is ignored
|
||||
(tested).
|
||||
|
||||
### Why no consent banner is needed — and why the analytics work is different
|
||||
|
||||
Under GDPR the lookup is **transient processing to deliver a service the user
|
||||
explicitly asked for**. The visitor pressed "Use my location" *wanting* local
|
||||
weather; when the browser can't supply it, deriving a coarse city from the
|
||||
connection they are already using to reach us is a proportionate substitute for
|
||||
the same, requested purpose. Nothing is stored, nothing is profiled, nothing is
|
||||
shared with a third party (the database is a file we host — no processor
|
||||
relationship, no international transfer, no disclosure to name in the policy).
|
||||
The data subject can override the result with one tap, and the result is
|
||||
explicitly labelled a guess rather than presented as a finding about them.
|
||||
|
||||
**ePrivacy Art. 5(3) is not engaged at all**, and this is the cleanest part of
|
||||
the argument. That article governs *storing information on, or gaining access to
|
||||
information stored in, a user's terminal equipment* — which is why cookies,
|
||||
localStorage and device fingerprinting need consent regardless of whether the
|
||||
data is personal. An IP address arrives in the request headers as an unavoidable
|
||||
property of routing a packet. We neither store anything on the device nor read
|
||||
anything from it. There is nothing for a consent banner to be about.
|
||||
|
||||
**Contrast with the analytics session-id work.** That one deliberately *writes
|
||||
an identifier onto the visitor's device* in order to correlate their actions
|
||||
across a session. That is squarely Art. 5(3) storage/access, it is not strictly
|
||||
necessary to deliver a service the user requested, and consent is therefore
|
||||
required — hence `consent.js` and the consent gate on that branch. The two
|
||||
features look adjacent ("both touch visitor data") and are legally opposite:
|
||||
one reads a routing header and forgets it, the other plants a durable
|
||||
identifier. Keep them separate in the code and in the privacy copy, and never
|
||||
let the geoip fallback be folded behind the analytics consent gate — that would
|
||||
imply consent is what makes it lawful, which is the wrong basis and would
|
||||
silently break the fallback for everyone who declines analytics.
|
||||
|
||||
### The privacy page follows the flag
|
||||
|
||||
`frontend/templates/privacy.html.j2` currently says, in the visitor's own
|
||||
words: *"Thermograph never looks up your location from your IP address."*
|
||||
Turning this feature on makes that sentence **false**, and a stale privacy
|
||||
statement is a worse outcome than not shipping the feature.
|
||||
|
||||
So the paragraph is switched on `THERMOGRAPH_GEOIP`, which both containers read
|
||||
from the same `/etc/thermograph.env` (`content.py`'s `geoip_enabled()`). The
|
||||
statement and the behaviour flip in the same deploy, and the frontend unit tests
|
||||
assert both branches. The frontend reads the flag for *nothing else* — it does
|
||||
not gate any behaviour on it.
|
||||
|
||||
---
|
||||
|
||||
## 6. Graceful degradation
|
||||
|
||||
Every one of these returns `None` → HTTP 204 → the client does exactly what the
|
||||
site does today. There is no failure mode worse than the status quo.
|
||||
|
||||
| Case | Handled where | Behaviour |
|
||||
| --- | --- | --- |
|
||||
| Flag unset | `enabled()` | 204 |
|
||||
| Flag set, no database file | `enabled()` (`os.path.isfile`) | 204 |
|
||||
| `maxminddb` not installed | lazy import in `_open()` | 204 |
|
||||
| Truncated / corrupt / half-written file | `_open()` try/except | 204 (tested) |
|
||||
| Private ranges (10/8, 172.16/12, 192.168/16) | `parse_ip` | 204 — every LAN-dev request |
|
||||
| Loopback, link-local, multicast, reserved, unspecified | `parse_ip` | 204 |
|
||||
| **CGNAT 100.64/10** | `parse_ip` + explicit `_EXTRA_RESERVED` | 204 — common on mobile carriers |
|
||||
| IPv6 (global) | `parse_ip` | looked up normally; MMDB covers v6 |
|
||||
| IPv6 ULA `fc00::/7`, doc `2001:db8::/32` | `_EXTRA_RESERVED` | 204 |
|
||||
| IPv4-mapped IPv6 `::ffff:a.b.c.d` | `parse_ip` unwraps | treated as the v4 address |
|
||||
| `1.2.3.4:5678`, `[v6]:443`, `%zone` suffixes | `parse_ip` | normalised |
|
||||
| Missing `X-Forwarded-For` | `_client_ip` falls back to peer | peer is loopback behind Caddy → 204 |
|
||||
| A whole XFF chain passed as one string | `parse_ip` | rejected (the route already takes the left-most hop) |
|
||||
| No record for the address | `lookup()` | 204 |
|
||||
| Record with a country but **no city** | `suggestion_from_record` | 204 — the VPN/proxy case |
|
||||
| Record whose stated radius > 100 km | `suggestion_from_record` | 204 |
|
||||
| Reader raises on a bad node | `lookup()` try/except | 204 |
|
||||
|
||||
The explicit `_EXTRA_RESERVED` list is not redundant with `ipaddress.is_global`:
|
||||
the exact membership of `is_private`/`is_global` has moved between CPython
|
||||
releases (100.64/10 in particular). Which Python the container happens to run
|
||||
must not decide whether we try to geolocate a carrier-NAT address.
|
||||
|
||||
---
|
||||
|
||||
## 7. The UI — presenting a guess as a guess
|
||||
|
||||
Copy shipped in `frontend/static/geoip.js`:
|
||||
|
||||
> **Showing weather near Rotterdam, South Holland.**
|
||||
> That is a rough guess from your internet connection, not from your device — it
|
||||
> can be tens of kilometres out. *IP Geolocation by DB-IP*
|
||||
>
|
||||
> **[ Not right? Choose your spot ]**
|
||||
|
||||
Design rules, and why each one:
|
||||
|
||||
1. **Named as a guess, in the same breath as the result.** Not a footnote, not a
|
||||
tooltip. City-level accuracy is ~50 km against a ~2 mile grid cell — the
|
||||
product's own unit of precision is 25× finer than the input.
|
||||
2. **The correction is a permanent, full-width, ≥44 px control**, not a
|
||||
dismissible toast. If the guess is wrong, fixing it must be the most obvious
|
||||
thing on screen.
|
||||
3. **A guess is never persisted.** `updateHash()` returns early while
|
||||
`approx` is true: no `localStorage` write, no URL hash. A guess that became
|
||||
sticky would be indistinguishable from a choice on the next visit, and a
|
||||
guessed URL would propagate a rough location the sender never picked when
|
||||
shared. Only an explicit pick is remembered.
|
||||
4. **The headings are rewritten while approximate.** The hero says "Roughly near
|
||||
Rotterdam" instead of "Today in …", and the results heading says "Near
|
||||
Rotterdam" with an "approximate — from your connection, not your device"
|
||||
tag. This matters more than it sounds: the guessed cell centre gets
|
||||
reverse-geocoded like any other point, which would otherwise name a
|
||||
*neighbourhood* — presenting street-level precision for a city-level input.
|
||||
5. **Visually recessed** — `--surface-2`, dashed border, muted note text. It
|
||||
reads as an aside, not as an answer.
|
||||
6. **It never pre-empts the permission prompt.** It runs *after* a
|
||||
declined/failed prompt, or on load only where the browser will not offer
|
||||
geolocation at all (insecure context) and nothing is remembered.
|
||||
7. **Never for a visitor who already has a place.** Theirs is better.
|
||||
|
||||
Attribution rides *with the result*, which is what CC BY 4.0 and DB-IP's own
|
||||
wording ask for ("on pages displaying results") — and it is derived from the
|
||||
installed file's metadata, so swapping databases changes the credit
|
||||
automatically.
|
||||
|
||||
### Interaction with `home.locate`
|
||||
|
||||
`home.locate` still means exactly what it means today: "someone tapped *Use my
|
||||
location*", whatever the outcome. It is unchanged, so the existing series stays
|
||||
comparable across this change. Two new allowlisted events sit beside it:
|
||||
|
||||
- `home.geoip_shown` — a suggestion was rendered. `home.locate` minus
|
||||
`home.geoip_shown` is roughly the population still hitting the dead end.
|
||||
- `home.geoip_fixed` — the visitor corrected the guess. This is the accuracy
|
||||
signal: a high ratio means the database is putting people in the wrong place
|
||||
and the feature is doing harm, which is the number to watch after rollout.
|
||||
|
||||
Both are aggregate counts through the existing beacon, with **no location
|
||||
dimension** — the suggestion itself is never recorded anywhere.
|
||||
|
||||
---
|
||||
|
||||
## 8. Database lifecycle
|
||||
|
||||
**On the host, refreshed by a systemd timer, bind-mounted read-only.** Not baked
|
||||
into the image.
|
||||
|
||||
- `infra/deploy/geoip-refresh.sh` — resolves the current month's DB-IP release
|
||||
(falling back to last month's, since the new file appears partway through the
|
||||
1st), downloads, gunzips, **verifies it opens and answers for a known public
|
||||
address**, and only then swaps it in atomically with `install`+`mv`. Exits 0
|
||||
on "already current". A failure leaves the existing file untouched — a failed
|
||||
refresh must never be worse than a stale one.
|
||||
- `geoip-refresh.timer` — `OnCalendar=Mon 04:17 UTC`, `RandomizedDelaySec=2h`,
|
||||
`Persistent=true`. Weekly rather than monthly even though DB-IP publishes
|
||||
monthly: a no-op run costs one HEAD request, and it means a missed or failed
|
||||
run self-heals within a week. (It also lands inside MaxMind's 30-day deletion
|
||||
window without further thought, should you switch.)
|
||||
- `infra/docker-compose.yml` mounts `${THERMOGRAPH_GEOIP_HOST_DIR:-/var/lib/thermograph/geoip}:/geoip:ro`
|
||||
into the backend. Read-only on purpose — the app must never be able to write it.
|
||||
- `geoip.py` re-opens when the file's `(mtime, size)` changes, so a refresh is
|
||||
picked up **without a restart or a deploy**.
|
||||
- `MODE_MMAP`: the 125 MB file costs page cache, not process RSS, and is shared
|
||||
across the four uvicorn workers on a host.
|
||||
|
||||
**Why not build-time.** It is a 125 MB data file on its own release cadence, not
|
||||
code. Baking it in adds 125 MB to every image pull, freezes freshness to
|
||||
whenever the last deploy happened, ties a data refresh to a code deploy, and
|
||||
(under GeoLite's EULA) cannot honour the 30-day deletion requirement on its own.
|
||||
|
||||
**Why not object storage.** We already have the bucket and could pull from it,
|
||||
but that adds credentials and a network dependency to fetch a file that is
|
||||
public and unauthenticated at the source. If a mirror is wanted later — to stop
|
||||
depending on DB-IP's uptime, or to pin exactly which release is live across the
|
||||
fleet — point `GEOIP_URL` at the bucket; the script needs no other change.
|
||||
|
||||
**Licence key storage.** With DB-IP: **none needed**, which is a large part of
|
||||
the recommendation. If MaxMind is chosen instead, the key goes in the SOPS vault
|
||||
(`infra/deploy/secrets/{prod,beta}.yaml`), renders into `/etc/thermograph.env`
|
||||
like every other secret, and `geoip-refresh.service` already picks it up via its
|
||||
`EnvironmentFile=` — set `GEOIP_URL` to the permalink that embeds it. Note the
|
||||
key would then be in a *URL*, so keep it out of any log line that echoes the
|
||||
URL: `geoip-refresh.sh` logs `$url`, and that line must be changed before using
|
||||
a key-bearing URL. **Flagged as a real footgun if you pick MaxMind.**
|
||||
|
||||
---
|
||||
|
||||
## 9. Interaction with caching / CDN
|
||||
|
||||
- **The homepage stays byte-identical for every visitor** — the container is
|
||||
server-rendered empty and hidden. `test_home_html_is_identical_regardless_of_client_address`
|
||||
asserts the body *and* the ETag match across different `X-Forwarded-For`
|
||||
values. That property is what keeps the homepage safe to cache, and it is the
|
||||
main reason the lookup is not in SSR.
|
||||
- **The endpoint is `no-store, private` + `Vary: *`** and answers 204 far more
|
||||
often than 200. Nothing about it is shareable between visitors.
|
||||
- **A CDN in front of the apex stays viable.** If Cloudflare (or a Caddy `cache`
|
||||
directive) is added later, the rule is one line: never cache `/api/*`. That is
|
||||
already the sane default and already how Caddy path-splits.
|
||||
- **The existing frontend caches are untouched.** `api_client.py`'s TTL/LRU
|
||||
cache never sees this request (it is browser→backend, not frontend→backend),
|
||||
and `static/cache.js`'s IndexedDB layer is keyed by `/cell` URLs. A guessed
|
||||
location produces ordinary `/api/v2/grade` traffic for an ordinary cell, so it
|
||||
warms and reuses the same caches as any other place — no new cache keyspace.
|
||||
- **One second-order effect worth naming:** guessed locations concentrate on
|
||||
city centroids, so they will warm the *same* handful of cells repeatedly.
|
||||
That is good for us (high cache hit rate, no extra Open-Meteo quota) but it
|
||||
does mean per-cell request counts stop being a clean proxy for distinct users
|
||||
near that cell.
|
||||
|
||||
---
|
||||
|
||||
## 10. Rollout
|
||||
|
||||
1. Merge with the flag off. Nothing changes — that is the point, and the tests
|
||||
assert it.
|
||||
2. On **beta only**: install the timer, run `geoip-refresh.sh` once by hand,
|
||||
confirm `/var/lib/thermograph/geoip/city.mmdb` exists and
|
||||
`curl -s -o /dev/null -w '%{http_code}' https://beta.thermograph.org/api/v2/geoip`
|
||||
still returns 204 (flag still off).
|
||||
3. Set `THERMOGRAPH_GEOIP=1` in beta's vault entry, redeploy **both** containers
|
||||
(backend for the lookup, frontend for the privacy copy).
|
||||
4. Verify by hand: decline the permission prompt; confirm the banner appears,
|
||||
the correction button works, the guess is *not* in `localStorage` and *not*
|
||||
in the URL; confirm `/privacy` now describes the fallback; confirm no
|
||||
`"cat":"geoip"` line and no matching IP appears in the access log.
|
||||
5. Watch `home.geoip_fixed / home.geoip_shown` for a week. If a large share of
|
||||
visitors are correcting the guess, the database is putting people in the
|
||||
wrong place — turn it off rather than tuning copy.
|
||||
6. Only then, prod.
|
||||
|
||||
Rollback is `THERMOGRAPH_GEOIP=0` and a redeploy; or just delete the database
|
||||
file, which disables it without touching config.
|
||||
|
||||
---
|
||||
|
||||
## 11. Decisions needed from the operator
|
||||
|
||||
1. **Which database?** Recommendation: **DB-IP IP to City Lite** (no account, no
|
||||
key, no retention clock, CC BY 4.0). The alternative is GeoLite2 City for
|
||||
`accuracy_radius` and weekly updates, at the cost of a vault-managed licence
|
||||
key and a 30-day deletion obligation. Code supports either unchanged.
|
||||
2. **Attribution placement.** Currently in the suggestion banner itself, beside
|
||||
the result. Alternatives: also on `/about`, or only on `/about`. DB-IP's
|
||||
stated wording asks for it "on pages displaying results", which the banner
|
||||
satisfies — confirm you're happy with a small credit line inside the hero.
|
||||
3. **SSR or client-side?** Recommendation: **client-side** (§4). Overruling this
|
||||
means accepting a location-varying homepage; if you do, put it on a separate
|
||||
non-canonical route, never on cached `/`.
|
||||
4. **Refresh cadence.** Currently weekly-with-no-op (§8). Alternatives: monthly
|
||||
to match DB-IP's release cadence exactly, or daily. Weekly is the
|
||||
self-healing middle.
|
||||
5. **Auto-suggest in insecure contexts?** Today the prototype suggests on load
|
||||
when the browser won't offer geolocation at all (LAN dev over plain HTTP) and
|
||||
nothing is remembered, since there is no prompt to decline. That is the only
|
||||
path where a lookup happens without an explicit user action. Prod is HTTPS so
|
||||
it never triggers there — but say if you'd rather it *never* happen without a
|
||||
tap, and it becomes a one-line change.
|
||||
6. **Threshold for calling it off.** Pick the `home.geoip_fixed / home.geoip_shown`
|
||||
ratio at which the feature gets turned off, *before* rollout rather than
|
||||
after — otherwise the number will always look explainable.
|
||||
|
||||
---
|
||||
|
||||
## 12. Things I think are bad ideas — flagged, not implemented
|
||||
|
||||
- **Don't put the lookup in the SSR homepage.** §4. A location-varying
|
||||
canonical page is a cache-poisoning and SEO hazard, and it geolocates the
|
||||
majority of visitors who already have a saved place and will never use it.
|
||||
- **Don't persist the guess.** Not to `localStorage`, not to the URL, not to an
|
||||
account preference. It is not a choice and must never become one silently.
|
||||
The branch enforces this in `updateHash()`.
|
||||
- **Don't use a third-party geolocation API**, even a "privacy-friendly" one.
|
||||
Sending visitor IPs to another company creates a processor relationship and a
|
||||
disclosure obligation, for a file we can just host. This was already the
|
||||
brief's requirement; restating it because it is the single most tempting
|
||||
shortcut here.
|
||||
- **Don't extend this to country-level personalisation** (units, language,
|
||||
default city) on the same lookup. Unit selection already works off the picked
|
||||
location's country code, and quietly widening an IP lookup from "a fallback
|
||||
the user asked for" to "we adapt the site to you" changes the legal analysis
|
||||
in §5 — it stops being strictly necessary for a requested service.
|
||||
- **Don't reuse the derived city anywhere it could be stored.** No metrics
|
||||
dimension, no audit field, no "popular cities" table fed from guesses. The
|
||||
moment a derived location is written next to anything timestamped, the
|
||||
privacy claim in §5 stops being true.
|
||||
- **Don't fold this behind the analytics consent gate.** §5. Different legal
|
||||
basis; gating it on consent both implies the wrong basis and breaks the
|
||||
fallback for everyone who declines analytics.
|
||||
- **Don't log `$url` once a MaxMind key-bearing permalink is in use** (§8).
|
||||
- **Be honest that this is a downgrade, not a feature.** A city-level guess is
|
||||
25× coarser than the grid the whole product is built on. It exists so a
|
||||
visitor who declined isn't stranded — not because IP location is good. If the
|
||||
correction rate says people don't want it, delete it rather than defend it.
|
||||
|
|
@ -118,6 +118,11 @@ def classify_inbound(path: str, base: str = "") -> str:
|
|||
# category keeps record_inbound from double-counting every interaction.
|
||||
if seg == "event":
|
||||
return "event"
|
||||
# Its own category so web/app.py's access-log middleware can skip it:
|
||||
# that route turns the client IP into a coarse location, so an access
|
||||
# line pairing the two on disk is exactly what the feature avoids.
|
||||
if seg == "geoip":
|
||||
return "geoip"
|
||||
return f"api:{seg}" if seg else "api:other"
|
||||
if p.endswith((".js", ".css", ".html", ".webmanifest", ".png", ".svg", ".ico",
|
||||
".json", ".woff", ".woff2", ".map", ".txt", ".xml")):
|
||||
|
|
@ -141,6 +146,13 @@ def classify_inbound(path: str, base: str = "") -> str:
|
|||
|
||||
EVENTS = frozenset({
|
||||
"home.locate", "home.digest_signup", "home.records_click", "home.share",
|
||||
# Approximate-location fallback (data/geoip.py). Pairs with home.locate:
|
||||
# that one counts taps on "Use my location" whatever the outcome, these two
|
||||
# count how often the IP fallback rescued a visitor who got no position
|
||||
# (geoip_shown) and how often the guess was wrong enough to correct
|
||||
# (geoip_fixed). Aggregate counts only — no location dimension, so the
|
||||
# suggestion itself is never recorded anywhere.
|
||||
"home.geoip_shown", "home.geoip_fixed",
|
||||
})
|
||||
# home.nav_{surface}: only these surfaces, so the key space stays bounded.
|
||||
NAV_SURFACES = frozenset({
|
||||
|
|
|
|||
330
backend/data/geoip.py
Normal file
330
backend/data/geoip.py
Normal file
|
|
@ -0,0 +1,330 @@
|
|||
"""Coarse IP -> approximate-location lookup, for visitors who don't share
|
||||
browser geolocation.
|
||||
|
||||
This is a *fallback suggestion*, never a position. Browser geolocation is
|
||||
metres-accurate; a city-level IP database is tens of kilometres accurate against
|
||||
a ~2 mile grid cell, so everything here is shaped so the caller cannot mistake
|
||||
one for the other: the payload is explicitly labelled approximate, and the UI
|
||||
that consumes it must offer a correction affordance (see frontend/static/geoip.js).
|
||||
|
||||
Design rules this module enforces, in order of importance:
|
||||
|
||||
1. **The IP is never persisted.** ``lookup()`` takes a string, reads a memory
|
||||
-mapped database, and returns a dict. Nothing is written anywhere — no DB
|
||||
row, no log line, no metric dimension. The web route deliberately keeps the
|
||||
geoip category out of the access log too (see web/app.py), so there is never
|
||||
a joinable (IP, derived-city) record on disk. Callers must not log the
|
||||
argument.
|
||||
|
||||
2. **The lookup is local.** The database is a file on our own disk in MaxMind's
|
||||
open MMDB format. No third-party API is called, so the visitor's IP never
|
||||
leaves the server and no data-processor relationship is created.
|
||||
|
||||
3. **Off unless deliberately turned on.** ``THERMOGRAPH_GEOIP`` must be truthy
|
||||
*and* ``THERMOGRAPH_GEOIP_DB`` must point at a readable file *and* the
|
||||
``maxminddb`` reader must be importable. Any one of those missing makes
|
||||
``enabled()`` False and every ``lookup()`` return None — which the route
|
||||
turns into a 204 and the client treats exactly like today's no-fallback
|
||||
behaviour. There is no failure mode here that is worse than the status quo.
|
||||
|
||||
4. **Silence beats a wrong guess.** A reserved/private/CGNAT/loopback address,
|
||||
an unparseable one, a record with no city (a country centroid would put a
|
||||
VPN user in the geographic middle of a country they aren't in), or a record
|
||||
whose stated accuracy radius is worse than
|
||||
``THERMOGRAPH_GEOIP_MAX_RADIUS_KM`` all return None rather than something
|
||||
plausible-looking.
|
||||
|
||||
Database choice is a deployment concern, not a code one: both DB-IP's
|
||||
*IP to City Lite* (CC BY 4.0, no account) and MaxMind's *GeoLite2 City*
|
||||
(GeoLite EULA, account + licence key) ship the same MMDB format with the same
|
||||
record shape, so either file works unchanged. ``attribution()`` reads the
|
||||
file's own metadata so the credit line the UI renders follows whichever file is
|
||||
installed. See GEOIP-APPROX-LOCATION.md for the comparison and the recommendation.
|
||||
"""
|
||||
import ipaddress
|
||||
import os
|
||||
import threading
|
||||
|
||||
from data import grid
|
||||
|
||||
FLAG_ENV = "THERMOGRAPH_GEOIP"
|
||||
DB_ENV = "THERMOGRAPH_GEOIP_DB"
|
||||
RADIUS_ENV = "THERMOGRAPH_GEOIP_MAX_RADIUS_KM"
|
||||
ATTRIB_ENV = "THERMOGRAPH_GEOIP_ATTRIBUTION"
|
||||
|
||||
# City-level databases claim ~50 km typical accuracy. Anything the file itself
|
||||
# admits is worse than this is a region/country centroid dressed up as a city,
|
||||
# which is exactly the "wrong part of the map" guess we'd rather not make.
|
||||
DEFAULT_MAX_RADIUS_KM = 100.0
|
||||
|
||||
# Extra reserved ranges checked explicitly rather than leaning on
|
||||
# ipaddress.is_global alone: the exact membership of is_private/is_global has
|
||||
# moved between CPython releases (100.64/10 in particular), and "which Python
|
||||
# is this container on" must not decide whether we try to geolocate a
|
||||
# carrier-NAT address.
|
||||
_EXTRA_RESERVED = (
|
||||
ipaddress.ip_network("100.64.0.0/10"), # RFC 6598 carrier-grade NAT
|
||||
ipaddress.ip_network("192.0.0.0/24"), # IETF protocol assignments
|
||||
ipaddress.ip_network("198.18.0.0/15"), # benchmarking
|
||||
ipaddress.ip_network("fc00::/7"), # IPv6 unique-local
|
||||
ipaddress.ip_network("2001:db8::/32"), # documentation
|
||||
)
|
||||
|
||||
_lock = threading.Lock()
|
||||
_reader = None
|
||||
_reader_key: "tuple | None" = None # (path, mtime, size) the open reader came from
|
||||
|
||||
|
||||
def _truthy(v: "str | None") -> bool:
|
||||
return (v or "").strip().lower() in ("1", "true", "yes", "on")
|
||||
|
||||
|
||||
def db_path() -> str:
|
||||
return os.environ.get(DB_ENV, "").strip()
|
||||
|
||||
|
||||
def max_radius_km() -> float:
|
||||
try:
|
||||
return float(os.environ.get(RADIUS_ENV, "") or DEFAULT_MAX_RADIUS_KM)
|
||||
except ValueError:
|
||||
return DEFAULT_MAX_RADIUS_KM
|
||||
|
||||
|
||||
def enabled() -> bool:
|
||||
"""True only when the flag is set AND a database file is actually present.
|
||||
Read on every call (not cached) so flipping the flag or dropping the file in
|
||||
doesn't need a restart, and so a test can toggle it with monkeypatch.setenv."""
|
||||
if not _truthy(os.environ.get(FLAG_ENV)):
|
||||
return False
|
||||
path = db_path()
|
||||
return bool(path) and os.path.isfile(path)
|
||||
|
||||
|
||||
def _open():
|
||||
"""The memory-mapped reader for the configured database, or None.
|
||||
|
||||
Re-opens when the file's (mtime, size) changes so the monthly refresh job
|
||||
can swap the file in atomically and be picked up without a deploy. mmap
|
||||
means the 100-200 MB file costs page cache, not process RSS, and is shared
|
||||
across uvicorn workers on the same host.
|
||||
"""
|
||||
global _reader, _reader_key
|
||||
path = db_path()
|
||||
if not path:
|
||||
return None
|
||||
try:
|
||||
st = os.stat(path)
|
||||
except OSError:
|
||||
return None
|
||||
key = (path, st.st_mtime_ns, st.st_size)
|
||||
with _lock:
|
||||
if _reader is not None and _reader_key == key:
|
||||
return _reader
|
||||
try:
|
||||
import maxminddb
|
||||
except ImportError:
|
||||
return None
|
||||
try:
|
||||
new = maxminddb.open_database(path, maxminddb.MODE_MMAP)
|
||||
except Exception: # noqa: BLE001 - a corrupt/half-written file must not 500
|
||||
return None
|
||||
old = _reader
|
||||
_reader, _reader_key = new, key
|
||||
if old is not None:
|
||||
try:
|
||||
old.close()
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
return new
|
||||
|
||||
|
||||
def reset() -> None:
|
||||
"""Drop the cached reader. For tests, and for anything that rewrites the
|
||||
database in place rather than swapping it atomically."""
|
||||
global _reader, _reader_key
|
||||
with _lock:
|
||||
r, _reader, _reader_key = _reader, None, None
|
||||
if r is not None:
|
||||
try:
|
||||
r.close()
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
|
||||
|
||||
def parse_ip(raw: "str | None") -> "ipaddress._BaseAddress | None":
|
||||
"""The routable address in ``raw``, or None.
|
||||
|
||||
Handles the shapes an X-Forwarded-For hop actually arrives in — a bare
|
||||
address, ``[v6]:port``, ``v4:port``, a ``%zone`` suffix — and rejects
|
||||
everything that can't identify a location on the public internet: private
|
||||
LAN ranges (every LAN-dev and internal-proxy request), loopback, link-local,
|
||||
multicast, carrier-grade NAT, IPv6 unique-local, and anything unparseable.
|
||||
An IPv4-mapped IPv6 address is unwrapped so ``::ffff:8.8.8.8`` is treated as
|
||||
the IPv4 address it is.
|
||||
"""
|
||||
s = (raw or "").strip()
|
||||
if not s:
|
||||
return None
|
||||
if s.startswith("["): # [2001:db8::1]:443
|
||||
s = s[1:].partition("]")[0]
|
||||
elif s.count(":") == 1: # 8.8.8.8:443 (a lone colon is never v6)
|
||||
s = s.partition(":")[0]
|
||||
s = s.partition("%")[0] # fe80::1%eth0
|
||||
try:
|
||||
addr = ipaddress.ip_address(s)
|
||||
except ValueError:
|
||||
return None
|
||||
mapped = getattr(addr, "ipv4_mapped", None)
|
||||
if mapped is not None:
|
||||
addr = mapped
|
||||
if not addr.is_global:
|
||||
return None
|
||||
if addr.is_private or addr.is_loopback or addr.is_link_local:
|
||||
return None
|
||||
if addr.is_multicast or addr.is_reserved or addr.is_unspecified:
|
||||
return None
|
||||
for net in _EXTRA_RESERVED:
|
||||
if addr.version == net.version and addr in net:
|
||||
return None
|
||||
return addr
|
||||
|
||||
|
||||
def _name(node) -> str:
|
||||
"""The English display name out of an MMDB names map. Both DB-IP Lite and
|
||||
GeoLite2 store ``{"names": {"en": ..., "de": ...}}``; DB-IP Lite is
|
||||
English-only in some builds, so fall back to whatever single name exists
|
||||
rather than dropping an otherwise-good record."""
|
||||
if not isinstance(node, dict):
|
||||
return ""
|
||||
names = node.get("names")
|
||||
if not isinstance(names, dict):
|
||||
return ""
|
||||
for key in ("en", "en-US"):
|
||||
v = names.get(key)
|
||||
if isinstance(v, str) and v.strip():
|
||||
return v.strip()
|
||||
for v in names.values():
|
||||
if isinstance(v, str) and v.strip():
|
||||
return v.strip()
|
||||
return ""
|
||||
|
||||
|
||||
def suggestion_from_record(record, *, max_radius: "float | None" = None) -> "dict | None":
|
||||
"""Turn one raw MMDB record into the wire payload, or None if it is too
|
||||
coarse to be worth showing. Split out from ``lookup()`` so every rejection
|
||||
rule is testable without a database file."""
|
||||
if not isinstance(record, dict):
|
||||
return None
|
||||
loc = record.get("location")
|
||||
if not isinstance(loc, dict):
|
||||
return None
|
||||
lat, lon = loc.get("latitude"), loc.get("longitude")
|
||||
if not isinstance(lat, (int, float)) or not isinstance(lon, (int, float)):
|
||||
return None
|
||||
if not (-90.0 <= lat <= 90.0) or not (-180.0 <= lon <= 180.0):
|
||||
return None
|
||||
|
||||
# No city name means a country (or continent) centroid. Showing one would
|
||||
# put every VPN user in the geographic middle of a country they may not be
|
||||
# in, which is worse than showing nothing: it looks like a real answer.
|
||||
city = _name(record.get("city"))
|
||||
if not city:
|
||||
return None
|
||||
|
||||
radius = loc.get("accuracy_radius")
|
||||
limit = max_radius_km() if max_radius is None else max_radius
|
||||
if isinstance(radius, (int, float)) and radius > limit:
|
||||
return None
|
||||
|
||||
subs = record.get("subdivisions")
|
||||
region = _name(subs[0]) if isinstance(subs, list) and subs else ""
|
||||
country_node = record.get("country") or record.get("registered_country") or {}
|
||||
country = _name(country_node)
|
||||
code = country_node.get("iso_code") if isinstance(country_node, dict) else None
|
||||
|
||||
# Snap to the app's own ~2 mile grid. This drops the database's fake
|
||||
# precision (city centroids are published to 4+ decimal places, which is
|
||||
# metres, for a value good to tens of kilometres) and returns exactly the
|
||||
# cell the weather would be fetched for anyway — so the coordinate carries
|
||||
# no more information than the city name already did.
|
||||
cell = grid.snap(float(lat), float(lon))
|
||||
|
||||
# "Springfield" alone is ambiguous; "Springfield, Illinois" is not. Region
|
||||
# first (more useful locally), country when there is no region.
|
||||
label = f"{city}, {region}" if region else (f"{city}, {country}" if country else city)
|
||||
|
||||
return {
|
||||
"approximate": True, # never omit: the client keys its copy off this
|
||||
"source": "ip",
|
||||
"label": label,
|
||||
"city": city,
|
||||
"region": region or None,
|
||||
"country": country or None,
|
||||
"country_code": code if isinstance(code, str) else None,
|
||||
"lat": cell["center_lat"],
|
||||
"lon": cell["center_lon"],
|
||||
"accuracy_radius_km": radius if isinstance(radius, (int, float)) else None,
|
||||
"attribution": attribution(),
|
||||
}
|
||||
|
||||
|
||||
def lookup(ip: "str | None") -> "dict | None":
|
||||
"""An approximate-location suggestion for ``ip``, or None.
|
||||
|
||||
None covers every degraded case — feature off, database missing, private or
|
||||
malformed address, no matching record, record too coarse — so the caller has
|
||||
exactly one branch to write and it is always "behave as if this feature did
|
||||
not exist". Never raises; never logs or stores ``ip``.
|
||||
"""
|
||||
if not enabled():
|
||||
return None
|
||||
addr = parse_ip(ip)
|
||||
if addr is None:
|
||||
return None
|
||||
reader = _open()
|
||||
if reader is None:
|
||||
return None
|
||||
try:
|
||||
record = reader.get(str(addr))
|
||||
except Exception: # noqa: BLE001 - a bad record must degrade, not 500
|
||||
return None
|
||||
if record is None:
|
||||
return None
|
||||
return suggestion_from_record(record)
|
||||
|
||||
|
||||
# --- attribution -----------------------------------------------------------
|
||||
# Both candidate databases are free *with attribution*, and each wants its own
|
||||
# wording, so the credit is derived from the installed file's own metadata
|
||||
# instead of hardcoded — swapping databases stays a config change. An explicit
|
||||
# THERMOGRAPH_GEOIP_ATTRIBUTION ("text|url") overrides, for a file whose
|
||||
# database_type we don't recognise.
|
||||
_ATTRIBUTIONS = {
|
||||
"dbip": ("IP Geolocation by DB-IP", "https://db-ip.com"),
|
||||
"geolite2": ("This product includes GeoLite data created by MaxMind",
|
||||
"https://www.maxmind.com"),
|
||||
}
|
||||
|
||||
|
||||
def attribution() -> "dict | None":
|
||||
"""``{"text", "url"}`` for the installed database, or None if it can't be
|
||||
determined. Rendered next to any result the visitor sees — both licences
|
||||
require credit on pages that display their data."""
|
||||
override = os.environ.get(ATTRIB_ENV, "").strip()
|
||||
if override:
|
||||
text, _, url = override.partition("|")
|
||||
return {"text": text.strip(), "url": url.strip() or None}
|
||||
reader = _open()
|
||||
if reader is None:
|
||||
return None
|
||||
try:
|
||||
kind = (reader.metadata().database_type or "").lower()
|
||||
except Exception: # noqa: BLE001
|
||||
return None
|
||||
if "dbip" in kind or "db-ip" in kind:
|
||||
text, url = _ATTRIBUTIONS["dbip"]
|
||||
elif "geolite" in kind:
|
||||
text, url = _ATTRIBUTIONS["geolite2"]
|
||||
else:
|
||||
return None
|
||||
return {"text": text, "url": url}
|
||||
|
|
@ -25,6 +25,13 @@ websockets==16.0
|
|||
# Recurring worker-tier jobs — city warming, IndexNow pings (see
|
||||
# notifications/scheduler.py). In-process; no broker/queue needed at this scale.
|
||||
apscheduler==3.11.3
|
||||
# MMDB reader for the optional approximate-location fallback (data/geoip.py).
|
||||
# The reader only, not MaxMind's geoip2 model layer — the raw dict record is all
|
||||
# we read, and staying on maxminddb keeps DB-IP's and MaxMind's files equally
|
||||
# usable. Imported lazily inside geoip.py so a missing wheel disables the
|
||||
# feature instead of breaking boot; pinned here so the flag is flippable on a
|
||||
# deployed image without a rebuild.
|
||||
maxminddb==2.8.2
|
||||
# The lake role's SQL engine (lake_app.py /query): embedded, parallel
|
||||
# partition-pruned scans over the ERA5 hive table. Only lake_app imports it.
|
||||
duckdb==1.4.2
|
||||
|
|
|
|||
263
backend/tests/data/test_geoip.py
Normal file
263
backend/tests/data/test_geoip.py
Normal file
|
|
@ -0,0 +1,263 @@
|
|||
"""Approximate-location fallback (data/geoip.py).
|
||||
|
||||
Every rejection rule is tested against a raw MMDB-shaped record rather than a
|
||||
real database file, so the suite stays hermetic and needs no 125 MB download —
|
||||
the record shape below is what both DB-IP IP-to-City Lite and MaxMind GeoLite2
|
||||
City actually return.
|
||||
"""
|
||||
import os
|
||||
|
||||
import pytest
|
||||
|
||||
from data import geoip
|
||||
from data import grid
|
||||
|
||||
|
||||
def _record(**over) -> dict:
|
||||
"""A well-formed city record. Rotterdam, as in the design doc's example."""
|
||||
rec = {
|
||||
"city": {"names": {"en": "Rotterdam"}},
|
||||
"country": {"iso_code": "NL", "names": {"en": "Netherlands"}},
|
||||
"subdivisions": [{"names": {"en": "South Holland"}}],
|
||||
"location": {"latitude": 51.9225, "longitude": 4.47917, "accuracy_radius": 20},
|
||||
}
|
||||
rec.update(over)
|
||||
return rec
|
||||
|
||||
|
||||
# --- address parsing --------------------------------------------------------
|
||||
@pytest.mark.parametrize("raw", [
|
||||
"8.8.8.8",
|
||||
"8.8.8.8:51234", # an XFF hop can carry the source port
|
||||
"2001:4860:4860::8888",
|
||||
"[2001:4860:4860::8888]:443",
|
||||
"::ffff:8.8.8.8", # IPv4-mapped IPv6 unwraps to the v4 address
|
||||
" 8.8.8.8 ",
|
||||
])
|
||||
def test_parse_accepts_routable_addresses(raw):
|
||||
assert geoip.parse_ip(raw) is not None
|
||||
|
||||
|
||||
@pytest.mark.parametrize("raw", [
|
||||
None, "", " ", "not-an-ip", "999.1.1.1", "8.8.8.8, 9.9.9.9", # a whole XFF chain
|
||||
"127.0.0.1", "::1", # loopback
|
||||
"10.0.0.4", "192.168.1.20", "172.16.5.5", # RFC 1918 — every LAN-dev request
|
||||
"100.64.7.9", # RFC 6598 carrier-grade NAT (mobile networks)
|
||||
"169.254.10.1", "fe80::1%eth0", # link-local
|
||||
"fd00::1", # IPv6 unique-local
|
||||
"2001:db8::1", # documentation range
|
||||
"224.0.0.1", # multicast
|
||||
"0.0.0.0", "::", # unspecified
|
||||
])
|
||||
def test_parse_rejects_everything_unroutable(raw):
|
||||
assert geoip.parse_ip(raw) is None
|
||||
|
||||
|
||||
def test_ipv4_mapped_v6_resolves_to_the_v4_address():
|
||||
assert str(geoip.parse_ip("::ffff:8.8.8.8")) == "8.8.8.8"
|
||||
|
||||
|
||||
# --- record -> suggestion ---------------------------------------------------
|
||||
def test_good_record_becomes_a_labelled_approximate_suggestion():
|
||||
s = geoip.suggestion_from_record(_record())
|
||||
assert s["approximate"] is True and s["source"] == "ip"
|
||||
assert s["label"] == "Rotterdam, South Holland"
|
||||
assert s["city"] == "Rotterdam" and s["country_code"] == "NL"
|
||||
|
||||
|
||||
def test_coordinates_are_snapped_to_the_app_grid_not_passed_through():
|
||||
"""The database publishes a city centroid to five decimal places — metre
|
||||
precision for a value good to tens of kilometres. We return the grid cell
|
||||
centre instead, so the payload carries no more precision than it has
|
||||
information."""
|
||||
s = geoip.suggestion_from_record(_record())
|
||||
cell = grid.snap(51.9225, 4.47917)
|
||||
assert (s["lat"], s["lon"]) == (cell["center_lat"], cell["center_lon"])
|
||||
assert s["lat"] != 51.9225
|
||||
|
||||
|
||||
def test_region_absent_falls_back_to_country_in_the_label():
|
||||
s = geoip.suggestion_from_record(_record(subdivisions=[]))
|
||||
assert s["label"] == "Rotterdam, Netherlands"
|
||||
|
||||
|
||||
def test_city_only_record_still_labels():
|
||||
s = geoip.suggestion_from_record(
|
||||
{"city": {"names": {"en": "Rotterdam"}}, "location": {"latitude": 1.0, "longitude": 2.0}})
|
||||
assert s["label"] == "Rotterdam"
|
||||
|
||||
|
||||
def test_non_english_only_names_are_still_usable():
|
||||
s = geoip.suggestion_from_record(_record(city={"names": {"nl": "Rotterdam"}}))
|
||||
assert s["city"] == "Rotterdam"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("rec", [
|
||||
None, {}, "garbage",
|
||||
{"location": {"latitude": 51.9, "longitude": 4.4}}, # country-only: no city
|
||||
{"city": {"names": {"en": "X"}}}, # no coordinates
|
||||
{"city": {"names": {"en": "X"}}, "location": {"latitude": 51.9}}, # half coordinates
|
||||
{"city": {"names": {"en": "X"}},
|
||||
"location": {"latitude": "51.9", "longitude": "4.4"}}, # wrong types
|
||||
{"city": {"names": {"en": "X"}},
|
||||
"location": {"latitude": 991.0, "longitude": 4.4}}, # out of range
|
||||
])
|
||||
def test_unusable_records_suggest_nothing(rec):
|
||||
assert geoip.suggestion_from_record(rec) is None
|
||||
|
||||
|
||||
def test_a_country_centroid_is_never_offered_as_a_city():
|
||||
"""The single most damaging failure mode: a VPN/proxy user gets a record
|
||||
with a country but no city, and we drop a pin in the middle of a country
|
||||
they are not in. Silence is the correct answer."""
|
||||
rec = _record()
|
||||
del rec["city"]
|
||||
assert geoip.suggestion_from_record(rec) is None
|
||||
|
||||
|
||||
def test_record_wider_than_the_accuracy_limit_is_rejected():
|
||||
assert geoip.suggestion_from_record(
|
||||
_record(location={"latitude": 51.9, "longitude": 4.4, "accuracy_radius": 500})) is None
|
||||
|
||||
|
||||
def test_record_with_no_stated_accuracy_is_accepted():
|
||||
"""DB-IP Lite omits accuracy_radius entirely; requiring it would disable the
|
||||
feature for that database."""
|
||||
s = geoip.suggestion_from_record(
|
||||
_record(location={"latitude": 51.9, "longitude": 4.4}))
|
||||
assert s is not None and s["accuracy_radius_km"] is None
|
||||
|
||||
|
||||
def test_accuracy_limit_is_configurable(monkeypatch):
|
||||
monkeypatch.setenv(geoip.RADIUS_ENV, "1000")
|
||||
assert geoip.suggestion_from_record(
|
||||
_record(location={"latitude": 51.9, "longitude": 4.4, "accuracy_radius": 500})) is not None
|
||||
|
||||
|
||||
# --- feature flag -----------------------------------------------------------
|
||||
def test_disabled_by_default(monkeypatch):
|
||||
monkeypatch.delenv(geoip.FLAG_ENV, raising=False)
|
||||
monkeypatch.delenv(geoip.DB_ENV, raising=False)
|
||||
assert geoip.enabled() is False
|
||||
assert geoip.lookup("8.8.8.8") is None
|
||||
|
||||
|
||||
def test_flag_without_a_database_stays_disabled(monkeypatch, tmp_path):
|
||||
monkeypatch.setenv(geoip.FLAG_ENV, "1")
|
||||
monkeypatch.setenv(geoip.DB_ENV, str(tmp_path / "missing.mmdb"))
|
||||
assert geoip.enabled() is False
|
||||
assert geoip.lookup("8.8.8.8") is None
|
||||
|
||||
|
||||
def test_database_without_the_flag_stays_disabled(monkeypatch, tmp_path):
|
||||
db = tmp_path / "city.mmdb"
|
||||
db.write_bytes(b"not really an mmdb")
|
||||
monkeypatch.delenv(geoip.FLAG_ENV, raising=False)
|
||||
monkeypatch.setenv(geoip.DB_ENV, str(db))
|
||||
assert geoip.enabled() is False
|
||||
|
||||
|
||||
def test_unreadable_database_degrades_to_no_suggestion(monkeypatch, tmp_path):
|
||||
"""A truncated or half-written file (a refresh job interrupted mid-copy)
|
||||
must behave like the feature is off, not raise."""
|
||||
db = tmp_path / "city.mmdb"
|
||||
db.write_bytes(b"not really an mmdb")
|
||||
monkeypatch.setenv(geoip.FLAG_ENV, "1")
|
||||
monkeypatch.setenv(geoip.DB_ENV, str(db))
|
||||
geoip.reset()
|
||||
assert geoip.enabled() is True
|
||||
assert geoip.lookup("8.8.8.8") is None
|
||||
geoip.reset()
|
||||
|
||||
|
||||
# --- lookup, end to end over a stand-in reader ------------------------------
|
||||
class _FakeReader:
|
||||
"""Stands in for maxminddb's reader: same tiny surface geoip.py uses."""
|
||||
|
||||
def __init__(self, by_ip, database_type="DBIP-City-Lite (compat=City)"):
|
||||
self.by_ip = by_ip
|
||||
self.asked = []
|
||||
self._type = database_type
|
||||
|
||||
def get(self, ip):
|
||||
self.asked.append(ip)
|
||||
return self.by_ip.get(ip)
|
||||
|
||||
def metadata(self):
|
||||
return type("M", (), {"database_type": self._type})()
|
||||
|
||||
def close(self):
|
||||
pass
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def live(monkeypatch, tmp_path):
|
||||
"""Feature on, with a stand-in reader — exercises enabled() + parse_ip() +
|
||||
suggestion_from_record() together without a real database file."""
|
||||
db = tmp_path / "city.mmdb"
|
||||
db.write_bytes(b"x")
|
||||
monkeypatch.setenv(geoip.FLAG_ENV, "1")
|
||||
monkeypatch.setenv(geoip.DB_ENV, str(db))
|
||||
reader = _FakeReader({"8.8.8.8": _record()})
|
||||
monkeypatch.setattr(geoip, "_open", lambda: reader)
|
||||
return reader
|
||||
|
||||
|
||||
def test_lookup_end_to_end(live):
|
||||
s = geoip.lookup("8.8.8.8")
|
||||
assert s["city"] == "Rotterdam"
|
||||
assert s["attribution"] == {"text": "IP Geolocation by DB-IP", "url": "https://db-ip.com"}
|
||||
|
||||
|
||||
def test_lookup_never_queries_the_database_for_a_private_address(live):
|
||||
assert geoip.lookup("192.168.1.20") is None
|
||||
assert live.asked == [] # rejected before the file is touched at all
|
||||
|
||||
|
||||
def test_lookup_returns_none_for_an_ip_with_no_record(live):
|
||||
assert geoip.lookup("9.9.9.9") is None
|
||||
|
||||
|
||||
def test_lookup_survives_a_reader_that_raises(live, monkeypatch):
|
||||
def boom(ip):
|
||||
raise ValueError("corrupt node")
|
||||
monkeypatch.setattr(live, "get", boom)
|
||||
assert geoip.lookup("8.8.8.8") is None
|
||||
|
||||
|
||||
def test_attribution_follows_the_installed_database(monkeypatch, tmp_path):
|
||||
db = tmp_path / "city.mmdb"
|
||||
db.write_bytes(b"x")
|
||||
monkeypatch.setenv(geoip.FLAG_ENV, "1")
|
||||
monkeypatch.setenv(geoip.DB_ENV, str(db))
|
||||
monkeypatch.setattr(geoip, "_open", lambda: _FakeReader({}, "GeoLite2-City"))
|
||||
assert "MaxMind" in geoip.attribution()["text"]
|
||||
|
||||
|
||||
def test_attribution_can_be_overridden_by_env(monkeypatch):
|
||||
monkeypatch.setenv(geoip.ATTRIB_ENV, "Data by Someone|https://example.org")
|
||||
assert geoip.attribution() == {"text": "Data by Someone", "url": "https://example.org"}
|
||||
|
||||
|
||||
def test_attribution_is_none_for_an_unrecognised_database(monkeypatch, tmp_path):
|
||||
db = tmp_path / "city.mmdb"
|
||||
db.write_bytes(b"x")
|
||||
monkeypatch.delenv(geoip.ATTRIB_ENV, raising=False)
|
||||
monkeypatch.setenv(geoip.DB_ENV, str(db))
|
||||
monkeypatch.setattr(geoip, "_open", lambda: _FakeReader({}, "Homegrown-City"))
|
||||
assert geoip.attribution() is None
|
||||
|
||||
|
||||
@pytest.mark.skipif(os.environ.get("THERMOGRAPH_GEOIP_DB", "") == "",
|
||||
reason="no real MMDB installed; set THERMOGRAPH_GEOIP_DB to run")
|
||||
def test_real_database_opens_and_answers():
|
||||
"""Opt-in smoke test against an actual installed database — the one thing
|
||||
the stand-in reader cannot prove (that our record-shape assumptions match a
|
||||
real file). Skipped in CI, run by hand after the refresh job first lands a
|
||||
file on a box."""
|
||||
import os as _os
|
||||
_os.environ[geoip.FLAG_ENV] = "1"
|
||||
geoip.reset()
|
||||
assert geoip.enabled()
|
||||
assert geoip.lookup("8.8.8.8") is not None or geoip.lookup("1.1.1.1") is not None
|
||||
geoip.reset()
|
||||
114
backend/tests/web/test_geoip_api.py
Normal file
114
backend/tests/web/test_geoip_api.py
Normal file
|
|
@ -0,0 +1,114 @@
|
|||
"""GET /api/v2/geoip — the approximate-location fallback endpoint.
|
||||
|
||||
The two properties worth locking down are behavioural, not cosmetic: it is
|
||||
inert when the flag is off (so shipping it disabled changes nothing), and it
|
||||
never writes the client IP anywhere.
|
||||
"""
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from core import audit
|
||||
from data import geoip
|
||||
from web import app as appmod
|
||||
|
||||
URL = "/thermograph/api/v2/geoip"
|
||||
|
||||
_SUGGESTION = {
|
||||
"approximate": True, "source": "ip", "label": "Rotterdam, South Holland",
|
||||
"city": "Rotterdam", "region": "South Holland", "country": "Netherlands",
|
||||
"country_code": "NL", "lat": 51.92028, "lon": 4.48339,
|
||||
"accuracy_radius_km": 20,
|
||||
"attribution": {"text": "IP Geolocation by DB-IP", "url": "https://db-ip.com"},
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
return TestClient(appmod.app)
|
||||
|
||||
|
||||
def _access_lines():
|
||||
lines = []
|
||||
for path in glob.glob(os.path.join(audit.ACCESS_DIR, "access-*.jsonl")):
|
||||
with open(path, encoding="utf-8") as fh:
|
||||
lines += [json.loads(ln) for ln in fh if ln.strip()]
|
||||
return lines
|
||||
|
||||
|
||||
def test_disabled_answers_204_with_no_body(client, monkeypatch):
|
||||
monkeypatch.delenv(geoip.FLAG_ENV, raising=False)
|
||||
r = client.get(URL, headers={"X-Forwarded-For": "8.8.8.8"})
|
||||
assert r.status_code == 204
|
||||
assert r.content == b""
|
||||
|
||||
|
||||
def test_enabled_hit_returns_the_suggestion(client, monkeypatch):
|
||||
monkeypatch.setattr(geoip, "lookup", lambda ip: dict(_SUGGESTION))
|
||||
r = client.get(URL, headers={"X-Forwarded-For": "8.8.8.8"})
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert body["approximate"] is True and body["label"] == "Rotterdam, South Holland"
|
||||
assert body["attribution"]["url"] == "https://db-ip.com"
|
||||
|
||||
|
||||
def test_response_is_never_cacheable_by_a_shared_cache(client, monkeypatch):
|
||||
"""This response is per-visitor by construction. A CDN or proxy that reused
|
||||
it would show one visitor another's city — the exact hazard that keeps this
|
||||
lookup out of the server-rendered homepage."""
|
||||
monkeypatch.setattr(geoip, "lookup", lambda ip: dict(_SUGGESTION))
|
||||
r = client.get(URL, headers={"X-Forwarded-For": "8.8.8.8"})
|
||||
assert "no-store" in r.headers["cache-control"]
|
||||
assert r.headers["vary"] == "*"
|
||||
|
||||
|
||||
def test_miss_answers_204(client, monkeypatch):
|
||||
monkeypatch.setattr(geoip, "lookup", lambda ip: None)
|
||||
r = client.get(URL, headers={"X-Forwarded-For": "8.8.8.8"})
|
||||
assert r.status_code == 204
|
||||
assert "no-store" in r.headers["cache-control"]
|
||||
|
||||
|
||||
def test_the_left_most_forwarded_hop_is_what_gets_looked_up(client, monkeypatch):
|
||||
seen = []
|
||||
monkeypatch.setattr(geoip, "lookup", lambda ip: seen.append(ip) or None)
|
||||
client.get(URL, headers={"X-Forwarded-For": "8.8.8.8, 10.0.0.1, 10.0.0.2"})
|
||||
assert seen == ["8.8.8.8"]
|
||||
|
||||
|
||||
def test_missing_forwarded_header_still_answers_cleanly(client, monkeypatch):
|
||||
"""No XFF (direct hit, LAN dev) falls through to the peer address, which is
|
||||
the loopback TestClient address — private, so no suggestion, no error."""
|
||||
monkeypatch.setenv(geoip.FLAG_ENV, "1")
|
||||
r = client.get(URL)
|
||||
assert r.status_code == 204
|
||||
|
||||
|
||||
def test_the_client_ip_is_not_written_to_the_access_log(client, monkeypatch):
|
||||
"""The whole point: reading an IP to derive a location must not leave a
|
||||
stored, joinable (IP, location) pair behind."""
|
||||
monkeypatch.setattr(geoip, "lookup", lambda ip: dict(_SUGGESTION))
|
||||
before = len(_access_lines())
|
||||
client.get(URL, headers={"X-Forwarded-For": "8.8.8.8"})
|
||||
after = _access_lines()
|
||||
assert len(after) == before
|
||||
assert not any(ln.get("ip") == "8.8.8.8" for ln in after)
|
||||
|
||||
|
||||
def test_a_normal_route_still_is_logged(client):
|
||||
"""Guards the exclusion above from silently widening to everything."""
|
||||
before = len(_access_lines())
|
||||
client.get("/thermograph/api/version")
|
||||
assert len(_access_lines()) > before
|
||||
|
||||
|
||||
def test_the_route_takes_no_parameters(client, monkeypatch):
|
||||
"""You cannot ask this endpoint about someone else's address: there is no
|
||||
input but the connection itself, and a supplied one is ignored."""
|
||||
seen = []
|
||||
monkeypatch.setattr(geoip, "lookup", lambda ip: seen.append(ip) or None)
|
||||
client.get(f"{URL}?ip=1.2.3.4", headers={"X-Forwarded-For": "8.8.8.8"})
|
||||
assert seen == ["8.8.8.8"]
|
||||
|
|
@ -21,6 +21,7 @@ from accounts import api_accounts
|
|||
from api import content_routes
|
||||
from core import audit
|
||||
from data import climate
|
||||
from data import geoip
|
||||
from notifications import digest
|
||||
from notifications import discord_bot
|
||||
from notifications import discord_interactions as discord_interactions_mod
|
||||
|
|
@ -306,7 +307,11 @@ async def revalidate_static(request, call_next):
|
|||
await run_in_threadpool(metrics.record_inbound, cat, response.status_code)
|
||||
# Retain per-request client IPs for later analysis; skip static assets and the
|
||||
# dashboard's own metrics polling to keep the log to real, meaningful traffic.
|
||||
if cat not in ("static", "metrics", "event"):
|
||||
# "geoip" is excluded for a different reason: that route derives a coarse
|
||||
# location FROM the client IP, so logging the two together would create the
|
||||
# one artefact this feature exists to avoid — a stored, joinable
|
||||
# (IP, approximate-location) pair. See api_geoip.
|
||||
if cat not in ("static", "metrics", "event", "geoip"):
|
||||
await run_in_threadpool(audit.log_access, {
|
||||
"ip": _client_ip(request), "method": request.method,
|
||||
"path": path, "status": response.status_code, "cat": cat})
|
||||
|
|
@ -809,6 +814,39 @@ async def api_event(request: Request) -> Response:
|
|||
return Response(status_code=204)
|
||||
|
||||
|
||||
async def api_geoip(request: Request) -> Response:
|
||||
"""An approximate location suggested from this request's own IP, for a
|
||||
visitor who has not shared browser geolocation. Feature-flagged OFF.
|
||||
|
||||
204 (no body) is the normal answer, and means "no suggestion" for every
|
||||
reason there can be: the flag is off, no database is installed, the address
|
||||
is private/reserved/unparseable, no record matched, or the record was too
|
||||
coarse to be worth showing. The client treats 204 exactly as it behaves
|
||||
today, so a disabled or broken lookup is indistinguishable from the feature
|
||||
never having shipped.
|
||||
|
||||
Privacy posture, deliberately:
|
||||
* the IP is read from this request's headers, used, and dropped — it is
|
||||
not written to the access log (see the middleware's category skip), any
|
||||
database, any metric dimension, or any log line here;
|
||||
* ``no-store`` + ``Vary: *`` so no shared cache or CDN can ever hand one
|
||||
visitor's suggestion to another — this is why the lookup is a separate,
|
||||
client-called endpoint rather than folded into the server-rendered
|
||||
homepage, whose HTML must stay byte-identical for everyone;
|
||||
* GET with no parameters and no body: there is nothing a caller can pass
|
||||
that would let them geolocate an address other than their own.
|
||||
"""
|
||||
# No RunAudit and no metrics dimension: both would record the derived
|
||||
# location, which is the sensitive half of the pair.
|
||||
hit = await run_in_threadpool(geoip.lookup, _client_ip(request))
|
||||
if hit is None:
|
||||
return Response(status_code=204, headers={"Cache-Control": "no-store"})
|
||||
return Response(
|
||||
content=json.dumps(hit), media_type="application/json",
|
||||
headers={"Cache-Control": "no-store, private", "Vary": "*"},
|
||||
)
|
||||
|
||||
|
||||
# --- API versioning --------------------------------------------------------
|
||||
# Non-backward-compatible additions land in a new version; every version stays
|
||||
# mounted and served simultaneously. v1 is the original contract (grade/geocode);
|
||||
|
|
@ -828,6 +866,7 @@ v2.add_api_route("/day", api_day, methods=["GET"])
|
|||
v2.add_api_route("/score", api_score, methods=["GET"])
|
||||
v2.add_api_route("/forecast", api_forecast, methods=["GET"])
|
||||
v2.add_api_route("/cell", api_cell, methods=["GET"])
|
||||
v2.add_api_route("/geoip", api_geoip, methods=["GET"], include_in_schema=False)
|
||||
v2.add_api_route("/metrics", api_metrics, methods=["GET"], include_in_schema=False)
|
||||
v2.add_api_route("/event", api_event, methods=["POST"], include_in_schema=False)
|
||||
|
||||
|
|
|
|||
|
|
@ -380,12 +380,27 @@ def about_page(request: Request) -> Response:
|
|||
return _respond_html(request, "about.html.j2", **ctx)
|
||||
|
||||
|
||||
def geoip_enabled() -> bool:
|
||||
"""Whether the backend's approximate-location fallback is switched on
|
||||
(backend/data/geoip.py). Read here for ONE reason: the privacy page's
|
||||
"Your location" section makes a factual claim about IP lookups, and a
|
||||
factual claim must not survive the behaviour it describes. Both containers
|
||||
read the same THERMOGRAPH_GEOIP out of the same /etc/thermograph.env, so
|
||||
the statement and the behaviour flip together in one deploy.
|
||||
|
||||
Nothing else in this repo branches on it — the interactive tool just calls
|
||||
the endpoint and gets a 204 when the feature is off.
|
||||
"""
|
||||
return os.environ.get("THERMOGRAPH_GEOIP", "").strip().lower() in ("1", "true", "yes", "on")
|
||||
|
||||
|
||||
def privacy_page(request: Request) -> Response:
|
||||
ctx = {
|
||||
"canonical_path": "/privacy",
|
||||
"breadcrumb": [("Home", f"{BASE}/"), ("Privacy", None)],
|
||||
"page_title": PAGES["privacy"]["title"],
|
||||
"page_description": PAGES["privacy"]["description"],
|
||||
"geoip": geoip_enabled(),
|
||||
}
|
||||
return _respond_html(request, "privacy.html.j2", **ctx)
|
||||
|
||||
|
|
|
|||
|
|
@ -7,10 +7,16 @@ import { initFindButton, setFindLabel } from "./mappicker.js";
|
|||
import { W, PW, H, PL, PR, plotTop, plotBot, xLabY, setChartWidth, chartPalette,
|
||||
tempChart, precipChart, dryChart, attachChartHover } from "./chart.js";
|
||||
import { track } from "./digest.js";
|
||||
import { fetchApprox, renderApprox, clearApprox } from "./geoip.js";
|
||||
import { TIER_COLORS, SCALE_TEMP, drynessColor, pctOrd, esc, todayISO, placeLabel,
|
||||
tierKeySegs, GUIDE_LINK, fmtPrecip, fmtWind, fmtHumid } from "./shared.js";
|
||||
|
||||
let selected = null; // {lat, lon}
|
||||
// True while `selected` came from the IP fallback rather than a real choice.
|
||||
// A guess is loaded (so the page is useful) but never remembered and never
|
||||
// written into the URL — see geoip.js for why that distinction matters.
|
||||
let approx = false;
|
||||
let approxLabel = ""; // the city the guess named, for headlines that must not overstate it
|
||||
|
||||
const dateInput = document.getElementById("date-input");
|
||||
const todayBtn = document.getElementById("today-btn");
|
||||
|
|
@ -32,8 +38,13 @@ todayBtn.addEventListener("click", () => {
|
|||
if (selected) runGrade();
|
||||
});
|
||||
|
||||
function selectLocation(lat, lon) {
|
||||
function selectLocation(lat, lon, opts) {
|
||||
selected = { lat, lon };
|
||||
// An approximate pick keeps its banner up; any real pick retires it, so the
|
||||
// "this is a guess" claim can never outlive the guess.
|
||||
approx = !!(opts && opts.approx);
|
||||
approxLabel = approx ? ((opts && opts.label) || "") : "";
|
||||
if (!approx) clearApprox(geoBox);
|
||||
// Show the raw coordinates immediately; render() replaces them with the resolved
|
||||
// neighborhood + city name once the grade fetch returns.
|
||||
locLabel.textContent = `${lat.toFixed(3)}, ${lon.toFixed(3)}`;
|
||||
|
|
@ -58,6 +69,7 @@ const hero = document.getElementById("hero");
|
|||
const heroLocate = document.getElementById("hero-locate");
|
||||
const heroPick = document.getElementById("hero-pick");
|
||||
const heroMsg = document.getElementById("hero-locate-msg");
|
||||
const geoBox = document.getElementById("geo-approx");
|
||||
|
||||
function locateMsg(text) {
|
||||
if (!heroMsg) return;
|
||||
|
|
@ -106,6 +118,16 @@ heroLocate?.addEventListener("click", () => {
|
|||
? "That took too long. Try again, or pick a spot on the map."
|
||||
: "Your location isn't available right now. Pick a spot on the map instead.",
|
||||
);
|
||||
// The dead end this fallback exists for. If the server can offer a
|
||||
// coarse guess, take over the message — telling someone to "pick a spot
|
||||
// on the map" while a guessed place is already loading reads as two
|
||||
// contradictory instructions.
|
||||
tryApprox().then((ok) => {
|
||||
if (!ok) return;
|
||||
locateMsg(err.code === err.PERMISSION_DENIED
|
||||
? "No problem — location permission stays off."
|
||||
: "Your device's location wasn't available.");
|
||||
});
|
||||
},
|
||||
{ enableHighAccuracy: false, timeout: 10000, maximumAge: 600000 },
|
||||
);
|
||||
|
|
@ -113,6 +135,30 @@ heroLocate?.addEventListener("click", () => {
|
|||
|
||||
heroPick?.addEventListener("click", () => { locateMsg(""); findBtn.click(); });
|
||||
|
||||
// ---- approximate-location fallback ----
|
||||
// Only ever a rescue for someone who has no location at all: it runs after a
|
||||
// declined/failed geolocation prompt, or on load in a context where the browser
|
||||
// will not offer geolocation at all (plain HTTP). It never runs for a visitor
|
||||
// who already has a place — theirs is better than any guess — and never
|
||||
// pre-empts the permission prompt.
|
||||
let approxTried = false;
|
||||
|
||||
async function tryApprox() {
|
||||
if (approxTried || selected) return false;
|
||||
approxTried = true;
|
||||
const data = await fetchApprox();
|
||||
// 204/anything unusable → exactly today's behaviour, nothing rendered.
|
||||
if (!data || selected) return false;
|
||||
renderApprox(geoBox, data, () => { locateMsg(""); findBtn.click(); });
|
||||
selectLocation(data.lat, data.lon, { approx: true, label: data.label });
|
||||
return true;
|
||||
}
|
||||
|
||||
// No geolocation available at all (insecure context, or a browser without it)
|
||||
// and nothing remembered: there is no prompt to decline, so offer the guess up
|
||||
// front rather than leaving the page with no place at all.
|
||||
if (!canLocate && !loadLastLocation()) tryApprox();
|
||||
|
||||
/** Re-point the hero card at the place the visitor actually chose. */
|
||||
function updateHero(data) {
|
||||
if (!hero) return;
|
||||
|
|
@ -121,7 +167,12 @@ function updateHero(data) {
|
|||
const g = targetDay && (targetDay.tmax || targetDay.tmin);
|
||||
if (!card || !g) return;
|
||||
|
||||
document.getElementById("hero-where").textContent = `Today in ${placeLabel(data)}`;
|
||||
// A guessed place is reverse-geocoded like any other cell, which would name a
|
||||
// *neighbourhood* — far more precision than an IP lookup ever had. Say
|
||||
// "roughly near <city>" instead, so the headline can't overstate what we know.
|
||||
document.getElementById("hero-where").textContent =
|
||||
approx ? `Roughly near ${approxLabel || placeLabel(data)}`
|
||||
: `Today in ${placeLabel(data)}`;
|
||||
document.getElementById("hero-band").textContent = g.grade;
|
||||
document.getElementById("hero-detail").textContent =
|
||||
`${targetDay.tmax === g ? "High temp" : "Low temp"} in the ${pctOrd(g.percentile)} percentile`;
|
||||
|
|
@ -267,7 +318,8 @@ function render(data) {
|
|||
results.innerHTML = `
|
||||
<div class="loc-head">
|
||||
<div class="loc-title">
|
||||
<h2>${placeLabel(data)}</h2>
|
||||
<h2>${approx ? `Near ${esc(approxLabel || placeLabel(data))}` : placeLabel(data)}</h2>
|
||||
${approx ? `<p class="loc-approx-tag">approximate — from your connection, not your device</p>` : ""}
|
||||
<ul class="loc-meta">
|
||||
${data.place ? `<li><b>${coords}</b></li>` : ""}
|
||||
<li>Climatology from <b>${yrs}</b></li>
|
||||
|
|
@ -605,6 +657,10 @@ function flashBtn(id, text) {
|
|||
|
||||
function updateHash() {
|
||||
if (!selected) return;
|
||||
// A guessed location is deliberately not remembered and not written into the
|
||||
// URL: next visit must ask again rather than treating a guess as a choice,
|
||||
// and a shared link must never carry a rough location the sender never picked.
|
||||
if (approx) return;
|
||||
history.replaceState(null, "", locHash(selected.lat, selected.lon, dateInput.value));
|
||||
saveLastLocation(selected.lat, selected.lon, dateInput.value);
|
||||
}
|
||||
|
|
|
|||
84
frontend/static/geoip.js
Normal file
84
frontend/static/geoip.js
Normal file
|
|
@ -0,0 +1,84 @@
|
|||
// Approximate-location fallback.
|
||||
//
|
||||
// The only reason this exists: a visitor who declines (or cannot use) browser
|
||||
// geolocation currently hits a dead end — search or the map picker, or nothing.
|
||||
// This asks the backend for a coarse guess derived from the request's own IP,
|
||||
// and offers it as a *suggestion*.
|
||||
//
|
||||
// Three rules the UI must never break, because the accuracy gap is enormous
|
||||
// (browser geolocation is metres; a city-level IP database is tens of
|
||||
// kilometres, against a ~2 mile grid cell):
|
||||
//
|
||||
// 1. Say it is a guess, in the visitor's own words, right next to the result.
|
||||
// Never present it as "your location".
|
||||
// 2. Put the correction one obvious tap away, permanently visible — not
|
||||
// hidden behind a dismissed toast.
|
||||
// 3. Never persist it. An explicit choice is remembered (localStorage) and
|
||||
// written into the URL hash; a guess is neither. A guess that became
|
||||
// sticky would be indistinguishable from a choice on the next visit, and
|
||||
// a guessed URL would propagate someone else's rough location when shared.
|
||||
//
|
||||
// The endpoint answers 204 for every "no" — feature off, no database, private
|
||||
// or unparsable address, no match, match too coarse — so there is exactly one
|
||||
// branch here and it is "behave exactly as the site does today".
|
||||
import { uv } from "./account.js";
|
||||
import { track } from "./digest.js";
|
||||
import { esc } from "./shared.js";
|
||||
|
||||
/**
|
||||
* Ask the backend for an approximate location.
|
||||
* Resolves to the suggestion object, or null when there is nothing to suggest.
|
||||
* Never rejects: a failed fetch is just "no suggestion".
|
||||
*/
|
||||
export async function fetchApprox() {
|
||||
try {
|
||||
const res = await fetch(uv("geoip"), { headers: { Accept: "application/json" } });
|
||||
if (res.status !== 200) return null; // 204 is the normal "no"
|
||||
const data = await res.json();
|
||||
if (!data || !data.approximate) return null; // never trust an unlabelled payload
|
||||
if (typeof data.lat !== "number" || typeof data.lon !== "number") return null;
|
||||
return data;
|
||||
} catch (e) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function attributionHtml(a) {
|
||||
// Both candidate databases are free *with attribution* and both ask for the
|
||||
// credit on the page that displays a result — so it rides with the result
|
||||
// rather than living only in a footer that this view doesn't have.
|
||||
if (!a || !a.text) return "";
|
||||
const text = esc(a.text);
|
||||
return a.url
|
||||
? ` <a class="geo-approx-credit" href="${esc(a.url)}" rel="noopener nofollow" target="_blank">${text}</a>`
|
||||
: ` <span class="geo-approx-credit">${text}</span>`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Render the suggestion banner into `host` and load the guessed place.
|
||||
*
|
||||
* @param host container element (hidden until there is something to show)
|
||||
* @param data a suggestion from fetchApprox()
|
||||
* @param onFix called when the visitor corrects the guess
|
||||
*/
|
||||
export function renderApprox(host, data, onFix) {
|
||||
if (!host) return;
|
||||
host.innerHTML =
|
||||
`<p class="geo-approx-line">Showing weather near <b>${esc(data.label)}</b>.</p>` +
|
||||
`<p class="geo-approx-note">That is a rough guess from your internet connection, ` +
|
||||
`not from your device — it can be tens of kilometres out.` +
|
||||
attributionHtml(data.attribution) + `</p>` +
|
||||
`<button type="button" class="btn-ghost geo-approx-fix">Not right? Choose your spot</button>`;
|
||||
host.hidden = false;
|
||||
host.querySelector(".geo-approx-fix")?.addEventListener("click", () => {
|
||||
track("home.geoip_fixed");
|
||||
onFix?.();
|
||||
});
|
||||
track("home.geoip_shown");
|
||||
}
|
||||
|
||||
export function clearApprox(host) {
|
||||
if (!host) return;
|
||||
host.hidden = true;
|
||||
host.innerHTML = "";
|
||||
}
|
||||
|
|
@ -1962,6 +1962,24 @@ details.hub-country > summary:hover { color: var(--accent); }
|
|||
.hero-jump a { color: var(--muted); text-decoration: none; }
|
||||
.hero-jump a:hover { color: var(--accent); }
|
||||
|
||||
/* Approximate-location banner (static/geoip.js). Read as an aside, not a
|
||||
result: recessed surface, no accent fill, and the correction button is a
|
||||
full-width ~44px target so it is never the hard thing to find on a phone. */
|
||||
.geo-approx {
|
||||
display: flex; flex-direction: column; gap: 8px;
|
||||
margin: 0; padding: 12px 14px; max-width: 42ch;
|
||||
background: var(--surface-2); border: 1px dashed var(--border); border-radius: 10px;
|
||||
}
|
||||
.geo-approx[hidden] { display: none; }
|
||||
.geo-approx-line { margin: 0; font-size: 14px; }
|
||||
.geo-approx-note { margin: 0; font-size: 12px; color: var(--muted); }
|
||||
.geo-approx-credit { color: var(--muted); text-decoration: underline; }
|
||||
.geo-approx-credit:hover { color: var(--accent); }
|
||||
.geo-approx .geo-approx-fix { min-height: 44px; font-size: 16px; width: 100%; }
|
||||
/* The same "this is a guess" claim on the results heading, so it survives a
|
||||
scroll past the hero. */
|
||||
.loc-approx-tag { margin: 2px 0 0; font-size: 12px; color: var(--muted); }
|
||||
|
||||
/* Desktop: copy beside the card rather than stacked above it. */
|
||||
@media (min-width: 900px) {
|
||||
.hero { grid-template-columns: 1.1fr 1fr; align-items: center; gap: 40px; }
|
||||
|
|
|
|||
|
|
@ -68,6 +68,13 @@
|
|||
<button type="button" id="hero-pick" class="btn-ghost">Pick on the map</button>
|
||||
</div>
|
||||
<p class="hero-locate-msg" id="hero-locate-msg" role="status" aria-live="polite" hidden></p>
|
||||
{# Approximate-location fallback (static/geoip.js). Server-rendered EMPTY
|
||||
and hidden on purpose: this page's HTML must stay byte-identical for
|
||||
every visitor so its ETag keeps validating and no shared cache can
|
||||
ever hand one person's city to another. The suggestion is fetched
|
||||
client-side, and only for a visitor who ended up with no location at
|
||||
all. #}
|
||||
<div class="geo-approx" id="geo-approx" role="status" aria-live="polite" hidden></div>
|
||||
<p class="hero-jump"><a href="#panel">See your week ↓</a></p>
|
||||
</div>
|
||||
</section>
|
||||
|
|
|
|||
|
|
@ -12,9 +12,27 @@
|
|||
there is very little about you to collect in the first place.</p>
|
||||
|
||||
<h2>Your location</h2>
|
||||
{# This section states a fact about what the server actually does, so it follows
|
||||
the THERMOGRAPH_GEOIP flag instead of being written once and left to rot.
|
||||
See content.py's geoip_enabled(). #}
|
||||
{% if geoip %}
|
||||
<p>The precise way the site learns where you are is if you press <b>Use my location</b>,
|
||||
which asks your browser for permission. You can decline, and picking a place on the map
|
||||
works just as well.</p>
|
||||
<p>If you decline, or your browser cannot offer it, we fall back to a rough guess at your
|
||||
city from your IP address so there is something local to show instead of nothing. It is
|
||||
always labelled as a guess, never as your position — it is accurate to tens of kilometres
|
||||
at best, and one tap corrects it. It is also not remembered: a guess is never saved in
|
||||
your browser or written into the page address, so only a place you actually pick is kept.</p>
|
||||
<p>That lookup runs entirely on our own server, against a database file we host ourselves.
|
||||
Your IP address is never sent to another company, and it is not stored: it is read from
|
||||
the request, used to find a city, and discarded. It is deliberately kept out of our
|
||||
request log for this feature too, so nothing anywhere links an address to a place.</p>
|
||||
{% else %}
|
||||
<p>Thermograph never looks up your location from your IP address. The only way the site
|
||||
learns where you are is if you press <b>Use my location</b>, which asks your browser for
|
||||
permission. You can decline, and picking a place on the map works just as well.</p>
|
||||
{% endif %}
|
||||
<p>Whichever way you choose a place, the coordinates are sent to the server only as part of
|
||||
the ordinary request for that grid square's climate data, and are not logged beyond the
|
||||
normal web-server request handling every site does. The place you picked is remembered in
|
||||
|
|
|
|||
58
frontend/tests/unit/test_geoip_ui.py
Normal file
58
frontend/tests/unit/test_geoip_ui.py
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
"""Approximate-location fallback, frontend side.
|
||||
|
||||
Two things are worth locking down here, and both are about honesty rather than
|
||||
mechanics: the server-rendered HTML must not vary by visitor (so no shared
|
||||
cache can leak one person's city to another), and the privacy page's factual
|
||||
claim about IP lookups must track the flag that decides whether they happen.
|
||||
"""
|
||||
import content
|
||||
|
||||
B = "/thermograph"
|
||||
|
||||
|
||||
def test_home_ships_an_empty_hidden_container(client):
|
||||
"""The suggestion is fetched client-side. If it were ever server-rendered
|
||||
into the homepage, that page's HTML — and its ETag — would vary per
|
||||
visitor, which is exactly the cache-poisoning shape this design avoids."""
|
||||
r = client.get(f"{B}/")
|
||||
assert r.status_code == 200
|
||||
assert 'id="geo-approx"' in r.text
|
||||
assert "hidden" in r.text.split('id="geo-approx"')[1].split(">")[0]
|
||||
|
||||
|
||||
def test_home_html_is_identical_regardless_of_client_address(client):
|
||||
"""Same bytes, same ETag, whatever the request claims about where it came
|
||||
from — the property that lets the homepage stay cacheable."""
|
||||
a = client.get(f"{B}/", headers={"X-Forwarded-For": "8.8.8.8"})
|
||||
b = client.get(f"{B}/", headers={"X-Forwarded-For": "1.1.1.1"})
|
||||
assert a.text == b.text
|
||||
assert a.headers["etag"] == b.headers["etag"]
|
||||
|
||||
|
||||
def test_privacy_says_no_ip_lookup_while_the_flag_is_off(client, monkeypatch):
|
||||
monkeypatch.delenv("THERMOGRAPH_GEOIP", raising=False)
|
||||
r = client.get(f"{B}/privacy")
|
||||
assert r.status_code == 200
|
||||
assert "never looks up your location from your IP address" in r.text
|
||||
|
||||
|
||||
def test_privacy_describes_the_fallback_when_the_flag_is_on(client, monkeypatch):
|
||||
"""Turning the feature on must not leave a published claim behind that is
|
||||
no longer true."""
|
||||
monkeypatch.setenv("THERMOGRAPH_GEOIP", "1")
|
||||
r = client.get(f"{B}/privacy")
|
||||
assert r.status_code == 200
|
||||
assert "never looks up your location from your IP address" not in r.text
|
||||
assert "rough guess" in r.text
|
||||
assert "not stored" in r.text
|
||||
|
||||
|
||||
def test_geoip_enabled_reads_the_shared_flag(monkeypatch):
|
||||
monkeypatch.delenv("THERMOGRAPH_GEOIP", raising=False)
|
||||
assert content.geoip_enabled() is False
|
||||
for on in ("1", "true", "YES", "on"):
|
||||
monkeypatch.setenv("THERMOGRAPH_GEOIP", on)
|
||||
assert content.geoip_enabled() is True, on
|
||||
for off in ("0", "", "no", "maybe"):
|
||||
monkeypatch.setenv("THERMOGRAPH_GEOIP", off)
|
||||
assert content.geoip_enabled() is False, off
|
||||
18
infra/deploy/geoip-refresh.service
Normal file
18
infra/deploy/geoip-refresh.service
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
[Unit]
|
||||
Description=Refresh the Thermograph IP-geolocation database
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
# GEOIP_URL / GEOIP_BASE / THERMOGRAPH_GEOIP_HOST_DIR can be overridden here if
|
||||
# the deployment switches databases. The leading `-` keeps a missing file
|
||||
# non-fatal, matching thermograph.service.
|
||||
EnvironmentFile=-/etc/thermograph.env
|
||||
ExecStart=/opt/thermograph/infra/deploy/geoip-refresh.sh
|
||||
# Downloading ~125 MB and verifying it needs no privilege beyond writing the
|
||||
# destination directory, and nothing else on the box.
|
||||
ProtectSystem=strict
|
||||
ReadWritePaths=/var/lib/thermograph/geoip
|
||||
PrivateTmp=yes
|
||||
NoNewPrivileges=yes
|
||||
90
infra/deploy/geoip-refresh.sh
Executable file
90
infra/deploy/geoip-refresh.sh
Executable file
|
|
@ -0,0 +1,90 @@
|
|||
#!/usr/bin/env bash
|
||||
# Refresh the IP-geolocation database used by the approximate-location fallback
|
||||
# (backend/data/geoip.py). Runs on the HOST (prod/beta), not in a container, on
|
||||
# a systemd timer — see geoip-refresh.timer.
|
||||
#
|
||||
# Why on the host and not baked into the image:
|
||||
# * it is a 100-200 MB data file on its own monthly release cadence, not code;
|
||||
# baking it in adds that to every image pull and freezes its freshness to
|
||||
# whenever the last deploy happened;
|
||||
# * MaxMind's GeoLite EULA *requires* deleting a database within 30 days of a
|
||||
# newer release, which a build-time copy cannot honour on its own;
|
||||
# * geoip.py memory-maps the file and re-opens it when (mtime, size) changes,
|
||||
# so an atomic swap here is picked up with no restart and no deploy.
|
||||
#
|
||||
# Default source: DB-IP's IP to City Lite, CC BY 4.0, no account and no licence
|
||||
# key — which is why this script needs no secret from the vault. Point
|
||||
# GEOIP_URL elsewhere (e.g. a MaxMind permalink carrying a licence key from
|
||||
# /etc/thermograph.env) to use a different database; geoip.py reads the file's
|
||||
# own metadata for the attribution line, so nothing else changes.
|
||||
#
|
||||
# sudo /opt/thermograph/infra/deploy/geoip-refresh.sh
|
||||
#
|
||||
# Exit status: 0 on a successful swap AND on "already current" (so the timer
|
||||
# doesn't alarm); non-zero only on a real failure, leaving any existing
|
||||
# database untouched — a failed refresh must never be worse than a stale one.
|
||||
set -euo pipefail
|
||||
|
||||
DEST_DIR="${THERMOGRAPH_GEOIP_HOST_DIR:-/var/lib/thermograph/geoip}"
|
||||
DEST="$DEST_DIR/city.mmdb"
|
||||
# DB-IP publishes one file per month at a predictable URL. Try this month, then
|
||||
# last month: the new file appears at some point on the 1st, so a timer firing
|
||||
# early would otherwise fail for a day.
|
||||
BASE="${GEOIP_BASE:-https://download.db-ip.com/free}"
|
||||
THIS_MONTH="$(date -u +%Y-%m)"
|
||||
LAST_MONTH="$(date -u -d "$(date -u +%Y-%m-01) -1 month" +%Y-%m)"
|
||||
|
||||
log() { echo "[geoip-refresh] $*"; }
|
||||
|
||||
mkdir -p "$DEST_DIR"
|
||||
tmp="$(mktemp -d "${TMPDIR:-/tmp}/geoip.XXXXXX")"
|
||||
trap 'rm -rf "$tmp"' EXIT
|
||||
|
||||
url=""
|
||||
if [ -n "${GEOIP_URL:-}" ]; then
|
||||
url="$GEOIP_URL"
|
||||
else
|
||||
for m in "$THIS_MONTH" "$LAST_MONTH"; do
|
||||
candidate="$BASE/dbip-city-lite-$m.mmdb.gz"
|
||||
if curl -fsSI --max-time 30 "$candidate" >/dev/null 2>&1; then
|
||||
url="$candidate"; break
|
||||
fi
|
||||
done
|
||||
fi
|
||||
[ -n "$url" ] || { log "no published database found at $BASE for $THIS_MONTH or $LAST_MONTH"; exit 1; }
|
||||
log "source: $url"
|
||||
|
||||
curl -fsSL --max-time 900 --retry 3 --retry-delay 10 -o "$tmp/db.gz" "$url"
|
||||
gunzip -c "$tmp/db.gz" > "$tmp/city.mmdb"
|
||||
|
||||
# Prove the file actually opens and answers before it can replace a working
|
||||
# one. A truncated download that merely *exists* would otherwise silently
|
||||
# disable lookups for a month.
|
||||
python3 - "$tmp/city.mmdb" <<'PY'
|
||||
import sys
|
||||
import maxminddb
|
||||
with maxminddb.open_database(sys.argv[1]) as r:
|
||||
meta = r.metadata()
|
||||
if "city" not in (meta.database_type or "").lower():
|
||||
sys.exit(f"unexpected database_type: {meta.database_type}")
|
||||
if r.get("8.8.8.8") is None and r.get("1.1.1.1") is None:
|
||||
sys.exit("database opened but answered nothing for known public addresses")
|
||||
print(f"[geoip-refresh] verified {meta.database_type} "
|
||||
f"built {meta.build_epoch} nodes={meta.node_count}")
|
||||
PY
|
||||
|
||||
if [ -f "$DEST" ] && cmp -s "$tmp/city.mmdb" "$DEST"; then
|
||||
log "already current: $DEST unchanged"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Atomic swap on the same filesystem: readers either see the whole old file or
|
||||
# the whole new one, never a half-written mmap. The app's open handle keeps the
|
||||
# unlinked inode alive until it re-opens on the mtime change.
|
||||
install -m 0444 "$tmp/city.mmdb" "$DEST.new"
|
||||
mv -f "$DEST.new" "$DEST"
|
||||
log "installed $DEST ($(stat -c %s "$DEST") bytes)"
|
||||
|
||||
# MaxMind's EULA requires deleting superseded copies within 30 days; keeping no
|
||||
# copies at all satisfies that trivially and is the right default for DB-IP too.
|
||||
find "$DEST_DIR" -maxdepth 1 -name 'city.mmdb.*' -type f -delete 2>/dev/null || true
|
||||
18
infra/deploy/geoip-refresh.timer
Normal file
18
infra/deploy/geoip-refresh.timer
Normal file
|
|
@ -0,0 +1,18 @@
|
|||
[Unit]
|
||||
Description=Monthly refresh of the Thermograph IP-geolocation database
|
||||
|
||||
[Timer]
|
||||
# Both candidate databases publish monthly. Weekly rather than monthly on
|
||||
# purpose: the script exits 0 without touching anything when the file is already
|
||||
# current, so the extra runs cost one HEAD request, and they mean a missed or
|
||||
# failed monthly run self-heals within a week instead of leaving the data a
|
||||
# month stale. (MaxMind additionally *requires* deleting a superseded copy
|
||||
# within 30 days, which a strictly-monthly timer would sit right on the edge of.)
|
||||
OnCalendar=Mon 04:17 UTC
|
||||
# Don't have every host in the fleet hit the mirror at the same second.
|
||||
RandomizedDelaySec=2h
|
||||
# Catch up after a reboot that spanned the scheduled time.
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
|
|
@ -207,3 +207,37 @@ THERMOGRAPH_BASE_URL=https://thermograph.org
|
|||
# required — Discord delivers content for mentions/DMs. Unset/0 => no gateway
|
||||
# connection. (Code: thermograph-backend notifications/discord_bot.py.)
|
||||
#THERMOGRAPH_DISCORD_BOT=1
|
||||
# --- approximate-location fallback (backend/data/geoip.py) ------------------
|
||||
# When a visitor declines (or cannot use) browser geolocation, suggest a coarse
|
||||
# city from their IP so the site has something local to show. OFF by default,
|
||||
# and inert until deploy/geoip-refresh.sh has landed a database at
|
||||
# /var/lib/thermograph/geoip/city.mmdb — with either missing, the site behaves
|
||||
# exactly as it does today (the endpoint answers 204 and the UI does nothing).
|
||||
#
|
||||
# Read by BOTH containers: the backend does the lookup; the frontend reads it
|
||||
# only to decide which paragraph /privacy renders about location. Turning this
|
||||
# on therefore changes the published privacy statement in the same deploy, on
|
||||
# purpose — do not set it in one place and not the other.
|
||||
#
|
||||
# The lookup is local (a file on our own disk, MaxMind MMDB format) — the
|
||||
# visitor's IP never leaves the server, so no third party becomes a data
|
||||
# processor. The IP is read in memory and discarded: it is written to no
|
||||
# database, no metric, and not even the access log (that route is excluded).
|
||||
#THERMOGRAPH_GEOIP=1
|
||||
# Where the refresh job writes the database on the host; bind-mounted read-only
|
||||
# into the backend container at /geoip. Only override if the default path is
|
||||
# unsuitable.
|
||||
#THERMOGRAPH_GEOIP_HOST_DIR=/var/lib/thermograph/geoip
|
||||
# Reject a record the database itself says is worse than this many km — that is
|
||||
# what stops a country-centroid guess being presented as a city. Default 100.
|
||||
#THERMOGRAPH_GEOIP_MAX_RADIUS_KM=100
|
||||
# Credit line shown beside any suggestion, as "text|url". Normally unset: it is
|
||||
# derived from the installed database's own metadata, so swapping DB-IP for
|
||||
# GeoLite2 changes the attribution automatically. Set it only for a database
|
||||
# whose metadata we don't recognise.
|
||||
#THERMOGRAPH_GEOIP_ATTRIBUTION=IP Geolocation by DB-IP|https://db-ip.com
|
||||
# Alternate download source for deploy/geoip-refresh.sh. Unset = DB-IP IP to
|
||||
# City Lite (CC BY 4.0, no account, no licence key — which is why nothing here
|
||||
# is a secret). A MaxMind permalink would embed a licence key and therefore
|
||||
# belongs in the SOPS vault rather than in this example file.
|
||||
#GEOIP_URL=
|
||||
|
|
|
|||
|
|
@ -129,6 +129,13 @@ services:
|
|||
# 503 and climate.py falls straight through to NASA — an accelerator,
|
||||
# never a dependency.
|
||||
THERMOGRAPH_LAKE_URL: http://lake:8141
|
||||
# Approximate-location fallback (backend/data/geoip.py). OFF unless
|
||||
# THERMOGRAPH_GEOIP is set truthy in /etc/thermograph.env, and inert even
|
||||
# then until deploy/geoip-refresh.sh has landed a database at the host
|
||||
# path bind-mounted below — so shipping the code changes nothing by
|
||||
# itself, and the mount being empty is a supported steady state.
|
||||
THERMOGRAPH_GEOIP: ${THERMOGRAPH_GEOIP:-}
|
||||
THERMOGRAPH_GEOIP_DB: /geoip/city.mmdb
|
||||
# Prod secrets live in /etc/thermograph.env: POSTGRES_PASSWORD,
|
||||
# THERMOGRAPH_AUTH_SECRET, THERMOGRAPH_VAPID_PRIVATE_KEY/_PUBLIC_KEY,
|
||||
# THERMOGRAPH_COOKIE_SECURE=1, mail/Discord keys, ... (see
|
||||
|
|
@ -145,6 +152,12 @@ services:
|
|||
# (above) points runtime state here instead, clear of the code.
|
||||
- appdata:/state
|
||||
- applogs:/app/logs
|
||||
# The IP-geolocation database, refreshed on the HOST by
|
||||
# deploy/geoip-refresh.sh (systemd timer, monthly) and mounted read-only:
|
||||
# the app must never be able to write it, and a 100-200 MB data file that
|
||||
# goes stale on its own schedule does not belong baked into the image.
|
||||
# geoip.py re-opens on mtime change, so a refresh needs no restart.
|
||||
- ${THERMOGRAPH_GEOIP_HOST_DIR:-/var/lib/thermograph/geoip}:/geoip:ro
|
||||
# No compose-level healthcheck override -- the image's own Dockerfile
|
||||
# HEALTHCHECK (port-aware via ${PORT}) already covers this, and frontend's
|
||||
# depends_on below reads it the same way.
|
||||
|
|
@ -208,6 +221,13 @@ services:
|
|||
PORT: 8080
|
||||
THERMOGRAPH_SERVICE_ROLE: frontend
|
||||
THERMOGRAPH_API_BASE_INTERNAL: http://backend:8137
|
||||
# Read for ONE purpose: which paragraph the /privacy page renders about
|
||||
# location. Frontend behaviour does not branch on it — the interactive
|
||||
# tool simply calls the backend endpoint and gets a 204 when the feature
|
||||
# is off. Same value as backend's, from the same /etc/thermograph.env, so
|
||||
# the published privacy statement can never drift from what the server
|
||||
# actually does.
|
||||
THERMOGRAPH_GEOIP: ${THERMOGRAPH_GEOIP:-}
|
||||
# No volumes -- stateless, holds no data of its own.
|
||||
cpus: ${FRONTEND_CPUS:-2}
|
||||
deploy:
|
||||
|
|
|
|||
Loading…
Reference in a new issue