Skip to content

A blank or junk country is not found, and the site tool's prefill finds every country the geocoder names - #80

Open
HughRunyan wants to merge 2 commits into
custom-location-runs-for-kosovofrom
a-blank-country-is-not-found
Open

HughRunyan wants to merge 2 commits into
custom-location-runs-for-kosovofrom
a-blank-country-is-not-found

Conversation

@HughRunyan

@HughRunyan HughRunyan commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Bug

_iso3_for (in city_params.py) turns a caller's country into the ISO3 code SWEET's defaults are keyed by. Its last resort is pycountry's fuzzy search, which scores every country for an empty name and scores a country for any of its subdivisions that match. So junk came back as a real country, silently:

input before why
"", " " GBR every country matches an empty name; the UK has the most subdivisions
"None" ITA Italy's province Frosinone contains "none"
"nan" THA Thailand's Nan province
"-", "0" FRA, SWE hyphenated / other subdivision matches
None, NaN AttributeError not LookupError, so callers crash instead of saying "not found"

The site tool's lat/lon prefill (sdst_prepopulate) names the country with Nominatim's English name, and had two more failures:

  • A point in no country crashed. At sea Nominatim can't place the point and geopy returns None; in Antarctica the address has no country. Both raised AttributeError.
  • 11 of Nominatim's country names were never found. "Turkey", "Ivory Coast", "Cape Verde", "East Timor", "Democratic Republic of the Congo", "Congo-Brazzaville", "Palestinian Territories", "Sahrawi Arab Democratic Republic", "Northern Cyprus", "Somaliland" and "South Ossetia" are spelled neither SWEET's nor pycountry's way, so each raised ValueError("Country '...' not found."). WasteMAP answered 500 for a custom site in Turkey.

Environment: SWEET_python custom-location-runs-for-kosovo (#77, 2b95a2b), pycountry 26.2.16, geopy 2.4.1, Python 3.13.

Reproduction:

from SWEET_python.city_params import City, _iso3_for
_iso3_for("")      # 'GBR'
_iso3_for("None")  # 'ITA'
City("*").dst_baseline_blank("", 2_000_000, 500.0, 25.0)  # builds the UK: 30,434 t CH4 in 2040

Expected: LookupError from _iso3_for, so dst_baseline_blank / cityparams_obj_for_blank_site raise ValueError("Country '' not found."); the prefill resolves every country Nominatim names, and says so where there is none.
Actual: the UK's model, Italy's, Thailand's; AttributeError; "Country 'Turkey' not found."

Fix

  1. No country is not found. _iso3_for raises LookupError for anything that isn't a non-blank string, and strips the name first (" Kosovo " was Serbia: SWEET's table missed the padded name and the fuzzy search found Kosovo-Metohija).
  2. The fuzzy answer counts only for the country's own name. It's taken only when the name is whole words of that country's name, official_name or common_name, or their initials ("UK", "UAE"). The best-ranked such country wins, so a subdivision can no longer carry a match. A province's own name is now not found even where the old answer was right ("England" → GBR).
  3. The prefill falls back to the geocoder's ISO code (_iso3_at): the English name first, then Nominatim's country_code. The name goes first because SWEET has its own codes for Aruba, Curaçao and Sint Maarten, which Nominatim codes as the Netherlands. Where there's no country, it raises ValueError("No country at <lat>, <lon>.").

Effect (model-output change for the prefill in Sint Maarten)

I reverse-geocoded 306 points through Nominatim with accept-language=en, as sdst_prepopulate does: one OpenStreetMap place=country label per country, plus hand-picked points for 81 territories and edge cases. That gave 222 distinct country names. I compared #77's lookup with this branch's, using Nominatim's own country_code as the reference:

count before after
names in SWEET's table / pycountry exact 207 resolved identical
fuzzy, resolved by own name (Brunei, São Tomé and Príncipe) 2 resolved identical
Abkhazia (code ge) 1 GEO via Georgia's subdivision GEO via the code
Sint Maarten (code nl) 1 NLD via the Netherlands' subdivision SXM (as Aruba → ABW, Curaçao → CUW already did)
the 11 names above 11 "not found" (WasteMAP 500) resolved via the code: TUR, CIV, CPV, TLS, COD, COG, PSE, ESH, CYP, SOM, GEO
no country (sea ×10, Antarctica) 11 points AttributeError "No country at …"
  • Sint Maarten's prefill becomes an open dump rather than a landfill, with food 48.2% rather than 35.0% and growth of 0.63% a year rather than 0.15%.
  • Breakaway regions Nominatim names as countries resolve to the country they're in, per its code: Abkhazia and South Ossetia → Georgia, Northern Cyprus → Cyprus, Somaliland → Somalia.
  • Every code and name in SWEET's own table resolves as before, so the city tool's Custom Location, which sends a code, is bit-identical for every country.
  • Climate TRACE's pipeline is unaffected. Only dst_baseline_blank, cityparams_obj_for_blank_site and sdst_prepopulate call _iso3_for, and all three are WasteMAP paths.

Junk after the change: "", " ", None, NaN, "None", "nan", "null", "undefined", "N/A", "-", "0", "*", "?", ".", "," all raise. Real inputs still resolve: "UK", "UAE", "CAR", "826", "GB", "MUS", "Niger", "United States", "Brunei", "Vatican", "Timor".

Tests

tests/test_country_lookup.py gains 41 cases, and 31 of them fail on #77:

  • blank/whitespace/non-string input raises LookupError
  • "None", "nan", "-", "0", "England", "Kurdistan" are not countries
  • padded names, initials, partial own names and numeric codes still resolve
  • dst_baseline_blank("") and cityparams_obj_for_blank_site(" ") raise "not found"
  • the prefill, with the geocoder and database faked, covers a point at sea, a point with no country, all 15 fallback cases, and an unknown name

The other 10 cases guard behaviour that already worked (Aruba, Curaçao, Abkhazia, "UK", the numeric code).

Full suite: 330 passed (Python 3.13 locally; CI runs 3.12).

Paired PR

RMI/WasteMAP#859, on the same branch name, so WasteMAP CI installs this branch. It makes the site tool's two endpoints answer 400 (not 500) when SWEET can't find the country. Its 4 new fast tests fail against #77 and pass against this branch. Merge #77 first, then this, then the WasteMAP PR.

Notes for review

Acceptance criteria

  • _iso3_for raises LookupError for "", whitespace, None, NaN and other non-strings.
  • _iso3_for("None"), ("nan"), ("-"), ("0") raise rather than resolve via a subdivision.
  • _iso3_for("MUS") == "MUS", "Niger" → NER, "United States" → USA (tests/test_population_growth.py).
  • Every code and name in SWEET's own table resolves as before.
  • The prefill resolves every country name Nominatim returned in the 306-point sweep, and raises ValueError (not AttributeError) for a point in no country.
  • Changelog updated (changelog/2026-10.md, changelog/README.md).

Definition of Done

Acceptance criteria met · tests & checks pass · docs (changelog) updated · reviewed & merged.

🤖 Generated with Claude Code

… the geocoder names

_iso3_for ended with pycountry's fuzzy search, which scores every country for
an empty name and scores a country for any of its subdivisions that match. So
"" and " " were the United Kingdom (the most subdivisions), "None" Italy
(Frosinone), "nan" Thailand (Nan province), and None raised AttributeError
rather than LookupError.

_iso3_for now raises LookupError for anything that isn't a non-blank string,
strips the name, and takes the fuzzy answer only when the name is whole words
of that country's own name or its initials ("UK", "UAE").

The site tool's prefill (sdst_prepopulate) crashed with AttributeError for a
point Nominatim can't place or gives no country, and failed for 11 English names
Nominatim uses that neither SWEET nor pycountry spell its way (Turkey, Ivory
Coast, Cape Verde, ...). _iso3_at tries the geocoder's name, then its ISO 3166-1
code, and the prefill says "No country at <lat>, <lon>." where there is none.

Over 306 reverse-geocoded points (222 country names): 210 resolve as before,
the 11 now resolve, Sint Maarten moves from NLD to SXM. Every code and name in
SWEET's own table resolves as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@HughRunyan HughRunyan added bug Something isn't working model-output-change Changes model OUTPUT values (expected progress, not breaking); results differ from prior runs labels Oct 9, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working model-output-change Changes model OUTPUT values (expected progress, not breaking); results differ from prior runs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant