You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat: timeseries schema v2 — published geography and sectorSegment - #974
Part of #915 (benchmark data v2 schema). First of three stacked PRs: schema → importer → download and plots.
What changes
Two row changes in the new src/schema/pathwayTimeseries.v2.json:
geography is the publication's own label, exactly as the pathway's metadata declares it: Global, a region key (e.g. ASEAN, Southeast Asia), or a declared ISO2 country. v1's closed enum couldn't hold those labels, so 7 of 16 files carried a label their metadata didn't declare, and the UI needed spelling folds (Data: IEA pathways spell the same region two ways across metadata and timeseries #945).
sectorSegment (required list) names the segments of the row's sector that the value covers. It uses the same vocabulary as data availability, but no sentinels, because a data row always covers a known segment. It replaces the per-metric "sector scope" (removed in the third PR).
Both rules depend on metadata in other files, so the new src/utils/validateTimeseries.ts enforces them from npm run schema:check.
Data
scripts/codemod-timeseries-v1-to-v2.ts migrated all 16 files. Values were checked identical before and after. Per file it changed only:
$schema
sectorSegment: ["Power generation"] on every row, which is what v1 meant for every Power metric
geography, via an explicit per-pathway table:
ACE South East Asia → ASEAN
IEA-APS/STEPS-2024 South East Asia → Southeast Asia
JRC's three files publish a South East Asia series that their metadata didn't declare. The region is now declared in the JRC metadata.
The member list (ID, MY, TH, VN plus the existing "Rest South East Asia" members, which include MN, KP and TW as JRC defines that region) is confirmed by @jacobvjk.
Checked
schema:check passes. On a scratch copy, a wrong label and a foreign segment are both reported.
ACE, IEA and JRC detail pages now plot their regional series with no fallback note. Checked locally in Chrome.
The generated index changes only by the relabelled geographies.
…rSegment
Two row changes (#915):
- geography is an open string that must be a label the pathway's
metadata declares: Global, a region key in the publication's own
wording, or a declared ISO2 country. v1's closed enum could not hold
labels such as "ASEAN" or "Southeast Asia", so files carried a label
their metadata did not declare and the UI needed spelling folds (#945).
- sectorSegment (required) lists the segments of the row's sector the
value covers, from the dataAvailability vocabulary but without its
sentinels: a data row always covers a known segment. It replaces the
per-metric sector scope.
Both rules need the metadata in other files, so validateTimeseries runs
them from schema:check. It shares "declared geography" with the
keyFeatures check by extracting declaredGeographies from
validateScopes, and reports a mistake repeated over many rows once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scripts/codemod-timeseries-v1-to-v2.ts changes three things per file
and nothing else (values verified identical):
- $schema becomes the v2 id
- every row gains sectorSegment ["Power generation"], which is what v1
meant: the taxonomy gave every Power metric that sector scope, and
every row is Power
- geography becomes the label the metadata declares, via an explicit
per-pathway table: ACE "South East Asia" -> "ASEAN", IEA-APS/STEPS
"South East Asia" -> "Southeast Asia"
JRC's three files publish a "South East Asia" series that their
metadata did not declare. Per Jacob, the region is declared instead
(members: ID, MY, TH, VN plus the "Rest South East Asia" members, to be
confirmed against GECO).
build-timeseries-files.ts accepts v2 files; the index changes only by
the relabelled geographies. ACE, IEA and JRC detail pages now plot
their regional series without a fallback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
JRC "South East Asia" confirmed by @jacobvjk: the region is declared in the three JRC metadata files with members ID, MY, TH, VN plus JRC's "Rest South East Asia" members (BN, KH, LA, MN, MM, KP, PH, SG, TW), as in this PR.
An empty list made validateTimeseries' geography check vacuously pass
and left the importer no pathway to place a new file next to; a repeated
id produced duplicate entries in the timeseries index (#974 review).
All 16 current files already comply.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the #975 review:
- A duplicate dataset id used to skip the file's remaining checks, so a
second problem in the same file only surfaced on the next run. Every
check now runs before the file is skipped.
- A new file is named after its id, so an id with "/" or a leading "."
could write outside the pathway's folder. Such ids are now an error.
- A missing metadata path is an explicit error rather than a fallback
to a guessed folder (unreachable today: unknown pathway ids are
rejected, and #974 now requires a non-empty pathwayId).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This comment says the validator runs from scripts/import-benchmark-data.ts, but that script does not exist in this change and the only production caller is scripts/schema-check-files.ts. Remove the future-caller claim so the documented enforcement path matches the current codebase.
v1 allowed an empty or repeated pathwayId list; v2 rejects both, and an
empty list would make relabel's "declared by every pathway" check pass
vacuously. The codemod now stops on either instead of writing an
invalid v2 file (#974 review). The 16 migrated files were unaffected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This module comment says the validator runs from scripts/import-benchmark-data.ts, but that script is not present in this PR and the only non-test caller is scripts/schema-check-files.ts. Remove the nonexistent integration claim until the importer PR adds it.
The reason will be displayed to describe this comment to others. Learn more.
from Codex:
P2 — pathwayTimeseries.v2.json (line 112): the parent technology enum excludes geothermal, hydrogen, and battery, while the power-specific branch permits them. JSON Schema combines both constraints, so those values are rejected for Power rows. I confirmed this with Ajv using geothermal. Add them to the parent enum (and regenerate types), or remove them from the power branch.
The reason will be displayed to describe this comment to others. Learn more.
Confirmed. This bug predates the PR (v1 has the same lists) and is tracked as #977. It is fixed in #979, which is stacked on this PR so the schema change itself stays reviewable. That fix:
makes the Power list exactly the Power technology keys in timeseriesTaxonomy.ts, which the plots and CSV download look every row up in
adds geothermal and energyStorage to the parent enum. Storage is energyStorage, per cookbook decision 0020, rather than battery.
drops battery and hydrogen. Hydrogen is not a Power technology in the taxonomy.
adds a schema test that keeps the two lists in agreement and checks that every Power value is in the parent enum, which is the AJV case you confirmed
No current data uses any of these values, so no data files change. Happy to fold #979 into this PR if you would rather review them together.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #915 (benchmark data v2 schema). First of three stacked PRs: schema → importer → download and plots.
What changes
Two row changes in the new
src/schema/pathwayTimeseries.v2.json:geographyis the publication's own label, exactly as the pathway's metadata declares it:Global, a region key (e.g.ASEAN,Southeast Asia), or a declared ISO2 country. v1's closed enum couldn't hold those labels, so 7 of 16 files carried a label their metadata didn't declare, and the UI needed spelling folds (Data: IEA pathways spell the same region two ways across metadata and timeseries #945).sectorSegment(required list) names the segments of the row's sector that the value covers. It uses the same vocabulary as data availability, but no sentinels, because a data row always covers a known segment. It replaces the per-metric "sector scope" (removed in the third PR).Both rules depend on metadata in other files, so the new
src/utils/validateTimeseries.tsenforces them fromnpm run schema:check.Data
scripts/codemod-timeseries-v1-to-v2.tsmigrated all 16 files. Values were checked identical before and after. Per file it changed only:$schemasectorSegment: ["Power generation"]on every row, which is what v1 meant for every Power metricSouth East Asia→ASEANSouth East Asia→Southeast AsiaJRC's three files publish a
South East Asiaseries that their metadata didn't declare. The region is now declared in the JRC metadata.The member list (ID, MY, TH, VN plus the existing "Rest South East Asia" members, which include MN, KP and TW as JRC defines that region) is confirmed by @jacobvjk.
Checked
schema:checkpasses. On a scratch copy, a wrong label and a foreign segment are both reported.🤖 Generated with Claude Code