Epic/v2 by AlexAxthelm · Pull Request #937 · RMI/tpr

AlexAxthelm · 2026-08-28T17:40:23Z

Combine #898, #935, #936 for preview of sledgehammer
Closes #945, #977, #968, #969, #970, #902

Foundation for epic #860 (best-effort / inheritance search): each of the 11 keyFeatures fields becomes an array of {sector, geography, value} entries so a pathway can hold different values for different parts of its coverage, and the #869 resolver can serve the most specific value for a search scope and fall back to broader ones. Also adds coreDrivers, dependencies, pathwayDescription and transitionAssessment. Additive only. v1 stays present and loadable, PathwayMetadataType still points at v1, and no data file changes here — the loader is repointed at v2 in a later commit once data carries the v2 $schema. Nothing consumes v2 yet.

Migrates the 4 ASEAN Centre for Energy and 3 IEA metadata files to pathwayMetadata.v2 via a new re-runnable codemod. src/data now holds 7 v2 and 49 v1 documents, which coexist because validateData routes each by its own $schema $id. Nothing reads v2 yet — the loader still points at v1, so the app is unchanged. All 7 resolve to a single widest-scope entry per keyFeature: cross-sector/South East Asia for ACE, cross-sector/Global for IEA. Both IEA files that carry pathwayOverview fold it into pathwayDescription as the lead paragraph; it has no readers in the app, so nothing observable moves. - scripts/codemod-v1-to-v2.ts splits v1's expertOverview into its three sections, wraps each keyFeature as one scoped entry, and scaffolds coreDrivers/dependencies. It skips files already on v2, so the remaining 49 are a re-run rather than a rewrite. A development tool only: there is no runtime v1 conversion, so un-migrated files simply will not load once the loader moves to v2. - The splitter accepts a bare line matching a section title as a heading. That exists for ACE-CNS-2024, whose "Core Drivers" heading lost its #### markers; without it, 1.5 KB of core-drivers prose folds into pathwayDescription and pushes it from 1204 to 2727 chars. - coreDrivers is scaffolded all-null per #858 rather than populated. The v1 "#### Core Drivers" prose does not map onto the 7 named fields mechanically: four paragraphs already exceed the 500-char cap, the italic labels ("Technology shifts", "Falling energy demand", "Economic growth") do not correspond 1:1 to the field names, and every section has unlabeled paragraphs with no destination. The codemod prints the prose it is not carrying so the hand-authoring ticket starts from the text. - transitionAssessment's maxLength goes 2500 -> 3000. 2500 was chosen for symmetry with pathwayDescription rather than measured; the longest section in the corpus is 2655 chars (ACE-RAS-2024), which made the codemod's own output invalid. pathwayDescription's 2500 is confirmed correct — the longest across all 56 files is 2459. Fixtures are added rather than converted, so the v1 fixtures stay v1 and the new coexistence tests can assert both halves. pathwayMetadata_v2_full carries several entries per field at different scopes, which #869 and #859 will need; _v2_minimal proves an all-empty keyFeatures document validates. One coexistence test documents a sharp edge deliberately: validateDataCollect filters entries to the single $id it is handed, so documents of the other version are dropped as neither valid nor invalid. That is what makes a mixed corpus work, and it is why repointing the loader has to report the count it skipped. Refs #858, #801. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Points the loader at pathwayMetadata.v2 and moves every consumer onto the scoped {sector, geography, value} shape. Only v2 documents load, so the app now shows the 7 migrated ACE/IEA pathways; the 49 still on v1 are skipped by $schema routing until they migrate. That skip is silent by construction — validateDataCollect drops non-matching documents as neither valid nor invalid — so pathwayMetadata.ts counts and logs them. Without it, 49 missing pathways look like a data bug. New src/utils/keyFeatureScope.ts answers "which entries apply to what the user is looking at": containment on both axes, where cross-sector means the union of the pathway's own declared sectors (not a universal match), and a geography scope contains a query when the query's ISO set is a subset of the entry's. Broader answers narrower, never the reverse. Deliberately no cost model, no ranking, no fallback — that is #869, and it is what will turn a non-match at the queried scope into a ranked broader-scope match rather than an exclusion. The emissionsTrajectory and policyAmbition facets now match like the sector and metric facets — ANY/ALL over a value list, empty list meaning absent — restricted to the entries whose scope contains the active sector/geography selection. The two near-identical 30-line arms collapse into one helper. concrete.includes(v) against an array is always false, so selecting either returned zero pathways, and option building emitted "[object Object]". Neither arm had any test coverage before — no filterPathways test passed either filter — which is why the whole suite stayed green while both were broken. Adding that coverage caught a regression that would otherwise have shipped: in v1 a missing field contributed undefined, which buildOptionsFromValues read as the absent bucket, but in v2 an empty entry array contributes no elements, so the "None" option disappeared from both dropdowns while the filter still honoured the token. Fixed with withAbsentOption, matching how the sector facet does it. Rendering keeps its current output. KeyFeatures reads through widestValue, a deliberately provisional stand-in for #869's resolver: it picks the value at the broadest declared scope, which reproduces v1 exactly for codemod-migrated data (one entry, at its widest scope). #859 replaces it and adds the badge naming the scope. PathwayDetailPage renders pathwayDescription and transitionAssessment under separate subheadings. v1's single expertOverview blob was three sections, so rendering only the description would have visibly dropped the Application to Transition Assessment text. The "Expert Overview" heading is left alone; naming is #859's call. Verified against the running app: 7 pathways load, the skip warning fires without error, and all 11 key features on IEA-NZE render values matching the source file — including the multi-select branch, which degrades silently rather than throwing when handed the wrong shape. Refs #858. Enables #869, #859. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Closes out #858's checklist. src/data/README.md described a format that no longer exists — and in the R example's case, one that never validated. Beyond the expected v1 leftovers (expertOverview, npm run json:check, a pbtar_schema.json link), the example used `name` as a bare string, the pre-#783 flat geography array, top-level publisher/publicationYear, and a `dataSource` field absent from every version of the schema. It would have failed against v1 as readily as against v2. Rewritten around v2: the two coexisting schema versions and the fact that only v2 documents are loaded, the scoped keyFeatures shape with its sentinels and the widest-scope rule, coreDrivers/dependencies/pathwayDescription/ transitionAssessment, the codemod for migrating an existing file, and the commands that actually exist. The R example is now verified rather than asserted: its blocks were extracted from this file, executed, and the resulting JSON validated against v2. Doing that corrected a wrong claim in an earlier draft — R's list() preserves NULL elements; the actual pitfall is jsonlite writing NULL as {}, which is why the helper passes null = "null". The validate_json R helper is dropped rather than repaired. The schema is split across common/*.json with absolute $refs that a single-URL jsonvalidate::json_validate() cannot resolve, so it documented a validation route that cannot succeed. Authors are pointed at npm run schema:check, which resolves the refs and additionally runs the cross-field scope checks that JSON Schema draft-07 cannot express. Tests: adds v2 counterparts to the existing v1 required-field cases — all 12 required fields, each of the 7 coreDrivers keys, unknown keys, and the dependencies enums. Also pins that pathwayDescription accepts null but not absence, the nullable-but-required distinction v2 relies on. The v1 REQ array still lists expertOverview on purpose: those cases validate v1 documents against v1, where it remains required. Finally, migrates the seven `keyFeatures: { emissionsTrajectory: "foo" }` stubs in ComparisonPage and PathwaySearch tests to the v2 shape. They kept their deliberately-invalid values, which exist to test degradation; the point is that a v1-shaped scalar in a v2 fixture silently exercises nothing. Refs #858. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

these didn't run past schema validation, so invalid selections were never caught

Selecting Sector=None alongside emissionsTrajectory or policyAmbition returned no pathways, even when a matching pathway existed. A pathway with no sectors holding "Significant decrease" at cross-sector/Global matched Sector=None on its own, but adding the keyFeature facet dropped it. The ABSENT/"None" token was reaching entriesInScope as though it were a scope. geographyScopeContains already ignored it, but sectorScopeContains compared it as a sector name, so no entry could ever match and valuesInScope came back empty — making the pathway look as if it held no values at any scope. The underlying mistake was treating the None bucket as a scope at all. It is a predicate about the pathway ("has no sectors" / "has no geography"), so it must not constrain which scope a value is read from. Fixed by stripping the token from both axes in entriesInScope rather than patching only the sector helper, so the two cannot disagree again; the guard inside geographyScopeContains stays for direct callers and is commented as such. No user-visible impact today: all seven currently-loaded pathways declare sectors, so nothing in the corpus could reach the broken path. It would have surfaced as soon as a sector-less pathway landed. Tests assert the symmetry rather than just the reported case, since the asymmetry is what caused the bug: the token is ignored on the sector axis, on the geography axis, and on both together; concrete tokens still narrow when combined with None, so the axis is not merely switched off; and the end-to-end filterPathways case is pinned, including that a value the pathway does not hold still correctly matches nothing. Reported by Copilot on PR #898. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…nment Combining a region geography filter with emissionsTrajectory or policyAmbition returned nothing for every region-scoped pathway. Verified on the real corpus: geography="Southeast Asia" kept ACE-ATS-2024 on its own, but adding that pathway's own emissionsTrajectory value ("Moderate increase") dropped it. All four ACE pathways were affected — the entire region-scoped half of the loaded corpus. The geography axis required the query's ISO set to be a subset of the entry's. That can essentially never hold, because the query vocabulary and each publication's region membership are separately maintained lists (#783): the filter token "Southeast Asia" carries 11 codes including TL, while ACE's own "South East Asia" carries 10 and omits it. One country's difference was enough to filter out every entry, leaving the field looking empty. It also put this layer at odds with filterPathways' own geography arm, which has always used overlap (isoSets.some(overlaps)). The two could therefore disagree about whether the same pathway matched the same region — which is why the bug only surfaced once a second filter was added. geographyScopeContains is now geographyScopeOverlaps: a non-empty intersection is a match. Renamed rather than changed behind the old name, since "contains" would no longer be true. Two behaviours are deliberately preserved: - "Global" stays a distinct predicate rather than expanding to every ISO code, mirroring the facet's `wantGlobal && pGlobal`. Under plain overlap a Global query would have matched every pathway. - An empty ISO set — an unrecognised token, or a region the publication never mapped — still matches nothing. Worth noting this needed care in the other direction: containment over an empty set is vacuously true and required an explicit size check, whereas overlap gets it right by construction. The sector axis keeps containment. It compares against a closed enum shared by every pathway, so there are no competing vocabularies to reconcile. Strict containment is still the right primitive for #869, where it drives ranking rather than acting as a hard filter. Tests: one expectation genuinely inverts — an entry scoped to TH now answers a "Southeast Asia" query — so that case was rewritten rather than deleted, since the old assertion encoded the bug. Added coverage for the two ways the vocabularies diverge (a publication label overlapping the query token, and the same label used as a query token matching nothing, because only "Southeast Asia" is in the filter vocabulary), plus a filterPathways-level assertion that the geography facet alone and the combined filter agree. That divergence was the defect, so it is now pinned directly. Found during a review pass over the PR. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Two entries at the same (sector, geography) carrying different values validated cleanly, then disagreed downstream. The schema's `uniqueItems` compares whole entries, so it only catches byte-identical duplicates — two entries differing solely in `value` are "unique" to the schema while being contradictory as data. Verified before the fix: a document with cross-sector/Global set to both "Significant decrease" and "Minor increase" passed npm run schema:check, after which widestValue returned the first by document order (so the detail page showed only that one) while valuesInScope returned both (so search matched the pathway under a value its own page does not display). validateScopedEntries now enforces one value per scope, per keyFeatures field, and names the entry a repeat collides with so the fix is obvious in a long list. Every repeat is reported and all point back at the first occurrence, rather than chaining 1->2, so a three-way collision reads as one problem. Rejecting rather than resolving is deliberate. Choosing a winner by document order would silently discard authored content, and the likely intent — an override of a broader scope — is not something the data can express. Better to fail loudly than to guess. The composite key separates its parts with \\u0000 so that ("Power", "SG") cannot collide with ("Power SG", ...); a test pins that, since a plain join would make the check quietly wrong for region labels containing the sector name. No effect on the current corpus — all 81 documents still validate. This matters now because hand-authoring finer scopes is the immediate next step, and adding a narrower override is exactly the edit that would have produced this. Found during a review pass over the PR. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The un-migrated-file count was an unconditional module-scope console.warn, so a production deploy printed "49 metadata file(s) still use schema v1 ... Migrate them with scripts/codemod-v1-to-v2.ts (#858)" to every visitor's console — naming an internal script and issue number. It also broke this module's own convention: assembleData logs only when a caller opts in via opts.warn, which this one does not. Gated on a new isViteDev() in loadData.ts, keyed off import.meta.env.DEV, which Vite statically replaces with false in production builds. isViteDev reuses an extracted readViteEnv() rather than re-implementing the env read, which is awkward enough to be worth having once: a try/catch around import.meta (typeof import breaks esbuild) plus a globalThis shim fallback. decideIncludeInvalid now shares it, with behaviour unchanged. It could not simply reuse decideIncludeInvalid, which answers a different question and returns false in dev unless VITE_INCLUDE_INVALID is set. Tests exercise the production path via vi.stubEnv("DEV", false) rather than the globalThis shim: vitest supplies a real import.meta.env, so readViteEnv prefers it and the shim is never consulted. Found during a review pass over the PR. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Three small fixes from Copilot's review pass. pathwayType's description read "Type of the pathway pathway." — inherited verbatim from v1, which the v2 generator copied. Fixed in both schemas rather than only v2, since leaving them divergent over a two-word correction helps nobody. policyAmbition's v2 description ran two sentences together: "...beyond currently legislated ones Scoped: see keyFeatures." Its v1 text is the only one of the eleven keyFeature descriptions that does not end in a period, so appending the scope note produced a run-on. Copilot flagged the line as another duplicated word; it is a missing period, and policyAmbition is the only field affected. schema-check-files.ts built GitHub annotation paths with join(r.dir, p.name), but p.name already carries the directory — getJsonFilesRecursive constructs it with join(base, d.name) — so annotations pointed at "src/data/src/data/foo.json" and resolved to nothing. Pre-existing, and not specific to the scope-check failures Copilot attributed it to: it affected every invalid file equally. It stayed hidden because it only surfaces when validation actually fails, which CI rarely sees. Verified by corrupting a file and confirming the emitted path resolves. Generated types and docs regenerated; the diffs are description-only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Any of the 31 technologies in technology.v1.json was legal under any of the 15 sectors in sector.v1.json: 465 pairings, of which the data uses 37. Jacob flagged the resulting generated type on #898 as "a bit of a random list ... not scoped to a sector". This constrains it. Enforced in validateScopedEntries rather than in the schema. #461 suggests mirroring the if/then sector conditional from pathwayTimeseries.v1.json, but that keyword pair defeats json-schema-to-typescript: the timeseries `data` items use exactly that shape and generate as `{ [k: string]: unknown }[]`. Applying it to sectors.items would collapse the `{ name; technologies }` object type and break every consumer of Sector, making the complaint worse rather than better. scopeSector.v2.json already documents the same tradeoff for the cross-sector constraint. Closed by default: a sector with no definition in timeseriesTaxonomy.ts accepts only an empty list. Letting undefined sectors through would mean the next data round populates technologies for a new sector and nothing checks them, which is the failure this exists to prevent. The error names the sector and says where to add its list. Only Power is populated, from the ten technologies POWER_SECTOR_DEFINITION already carries, and the allowlist is derived from it rather than duplicated so adding a sector is a single edit. The other 14 sectors' lists are content for the data round, not this PR. technologyBelongsToSector is tri-state ("yes" | "no" | "unknown") rather than boolean: a boolean would answer false for the 14 undefined sectors, indistinguishable from a real rejection. Validation treats unknown as a failure, while #869's technology axis can treat it as a match instead of silently dropping every non-Power pathway. The codemod reports offenders and leaves them in place. Deleting a technology someone recorded on purpose, or inventing a sector's taxonomy, would both be worse than a line in the report plus a refusal from schema:check. One fixture line changes: pathwayMetadata_v2_full.json gave Steel ["Hydrogen Use"], the only document in the repo the rule rejects. All 72 files in src/data pass untouched, Power being the only sector any of them populates. schema:check reports 81 valid / 0 invalid; 668 tests pass, up from 634. tsc --build goes from 245 to 247 errors, both the TS5097 forced by the required .ts import extension — loadData.ts:1 carries the identical error for the identical reason, and tsc is not a CI gate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Data availability has been derived, not authored: availabilityFor in searchUtils.ts guesses "Download" | "Link" | "Unavailable" from whether the timeseries index has an entry for the pathway and whether any publication link is described as "data". One coarse value per pathway, which cannot answer the question users actually have -- for this metric, in this sector, at what resolution, and can I get it without paying? This adds the stored form and its validation. The table UI is not here: it is specified to sit under the #872 filter bar, which is still open, and the example values are still pending. Landing the schema first unblocks the content authoring, which is the critical path. availabilityFor is left untouched -- it backs a live search facet, and retiring it while dataAvailability is authored for 0 of 72 files would regress search for every pathway. dataAvailability is optional, so existing files stay valid and authoring is incremental: { overall, byMetric[] }, one row per (metricName, sector, sectorSegment, geography). The object wrapper exists because #870's "Overall" row is a summary of the hosted timeseries file plus anything that does not fit the per-metric rows, and the array has nowhere to put free text. Rows carry a scopeGeography.v2 token, the same one keyFeatures uses, so entriesInScope/geographyScopeOverlaps can scope the table when the UI lands. #870's field list has only geographyCoverage, which is a coverage class rather than a scope, leaving the "respects the #872 selection" criterion with no field to match on. Two new common schemas: sectorSegment.v1.json, and dataAvailability.v1.json for the vocabularies with no other home. dataFormat splits into dataFormat + access rather than #870's single enum with parenthetical free|paywall -- the acceptance criteria want the tables/text and free/paywall indicators as separate signals, and a flat five-member enum would make the table string-match to recover them. Most of the work was already done by #461. metricName, sectorSegment and granularity are all sector-conditional, so timeseriesTaxonomy's technology helpers generalize to three axes behind vocabularyFor and membership; granularity reuses technologyBelongsToSector outright, since technology list. The tri-state is now TaxonomyMembership, with TechnologyMembership kept as an alias so #461's call sites do not churn. Eight cross-field checks join their siblings in validateScopedEntries, gated by schema:check and CI: declared sector, resolvable geography, metric the pathway reports, metric of the sector, segment of the sector, granularity of the sector, access null iff dataFormat is "In tool", and one row per scope. That last one matters because uniqueItems compares whole entries, so two rows agreeing on the scope and disagreeing on everything else validate cleanly and then have one cell to render in. Two things the plan got wrong, both found by building it: The metric axis is open rather than closed by default. Technologies and segments have no other constraint, so closing them is the only thing between a typo and production. metricName already has one -- it must appear in the pathway's own `metric` array -- so closing it too would add no safety while making dataAvailability unauthorable for the fourteen sectors whose metrics nobody has defined, which is the blockage UNSEGMENTED exists to avoid. Segments and granularity stay closed. A sector defining an *empty* vocabulary produced "(allowed: )", which reads as a bug in the checker rather than an answer. Now "-- that sector defines no metrics.", via a shared allowedClause helper covering all three cases; #461's sectors[].technologies message had the same defect and is fixed with it. Note the sector/metric check cannot currently reject real data: Power defines all five members of metric.v1.json's enum and every other sector is "unknown". It goes live when a second sector defines metrics. Tested directly, with a test pinning the reason. schema:check stays 81 valid / 0 invalid; 718 tests pass, up from 668; schema:generate is idempotent; tsc --build holds at its 247-error baseline. Each of the eight rules was confirmed to reject a broken copy of the fixture with its own message naming the offender. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Add scripts/import-pathway-data.ts, which reads the working Excel workbook (pathway_data_prepared.xlsx) and writes real v2 pathway metadata JSON, superseding the one-shot codemod-v1-to-v2.ts (#858). It reads the draft_* sheets (parameterized via SHEET_PREFIX and the --sheet-prefix flag, ready for the final pathway_* sheets), maps each row to its target file by publisher + scenario code + country, and emits scoped keyFeatures, coreDrivers, dependencies, and geography (with region->ISO memberships) that satisfy schema:check and validateScopedEntries. IEA WEO-2025 rows create new files and keep the 2024 ones; every other row updates its file in place. Anything that cannot be mapped cleanly is reported rather than written, so the run never produces schema-invalid output. Adds exceljs as a dev dependency. The regenerated data files are committed separately. Part of #860. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Split the single scrolling pathway detail page into a four-tab layout (At a glance / Overview / Timeseries / Scope & Granularity) per Jacob's wireframe, with a sticky sector + tab bar that persists on scroll. - Add an accessible Tabs/TabPanel primitive (WAI-ARIA tabs pattern: roving tabindex, Arrow/Home/End keys, aria-selected/controls wiring). - Back the active tab with a `?tab=` query param via a useActiveTab hook so tabs are deep-linkable, shareable, and survive reload/back. The URL mechanism is isolated behind the hook for a cheap later swap to path segments. - Move existing v1 content into tabs: At a glance (Expert Overview, plot, Key Features, coverage panels), Timeseries (Supplemental Info + plot). - Render labeled placeholders for the wireframe sections that need the metadata-v2 schema (#898): Assumptions & Trends, Dependencies, and the Data Availability table. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ity tab

Render the pathway's dependencies (conditions its outcomes rely on) as a table in the Overview tab, replacing the placeholder. Columns: Dependency, Evidence type (a strength-colored badge), and Constraint. A Sector column is shown only when the rows span more than one sector, so single-sector pathways match the three-column wireframe. An empty dependency list renders an explanatory empty state, since authoring is incremental. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Render coreDrivers as the Assumptions & Trends Overview in the Overview tab, replacing the last placeholder: one card per driver (Policies, Emissions targets, Technology costs, Investment, Macroeconomic drivers, Behavioral shifts, Other drivers). A null driver is shown as an explicit "not a core driver for this pathway" note rather than hidden, matching the schema's distinction between an absent driver and an undescribed one. Drops the now-unused SectionPlaceholder. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

A Power row's technology must satisfy both the all-sector enum and the Power list in the schema's if/then. The Power list had geothermal, battery and hydrogen, which the all-sector enum lacked, so they could never validate (#977). The Power list is now exactly the taxonomy's Power technology keys, which the plots and CSV download resolve every row against: geothermal and energyStorage (cookbook decision 0020; matches the Energy storage segment) are added to the all-sector enum, battery and hydrogen are dropped. A schema test keeps the two lists in agreement. No current data uses any of the changed values. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Timeseries now carry the metadata's own geography labels, and validateTimeseries enforces it (#974), so three workarounds for "South East Asia" vs "Southeast Asia" are no longer needed: - resolveGeography's labelVariant arm - regionMemberCodes' fallback to canonicalGeographyKey - hasGeography's label and spelling-fold attempts; it is now a plain lookup of the metadata token canonicalGeographyKey stays: comparisonScope uses it to spot two publishers using one region name with different members, which is a cross-publication question, not a data mismatch. The tests for the removed behaviour now assert the new contract: a different spelling is a different geography. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

An empty list made validateTimeseries' geography check vacuously pass and left the importer no pathway to place a new file next to; a repeated id produced duplicate entries in the timeseries index (#974 review). All 16 current files already comply. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>