Skip to content

feat(referential): stations referential exporter, format 1 (v1.0.0) - #2

Merged
MaxLeb merged 2 commits into
mainfrom
feat/export-referential
Sep 27, 2026
Merged

MaxLeb merged 2 commits into
mainfrom
feat/export-referential

Conversation

@MaxLeb

@MaxLeb MaxLeb commented Sep 27, 2026

Copy link
Copy Markdown
Owner

What

A new, stable part of the repository: export-referential and prim_api.referential. From five IDFM Opendatasoft datasets, it builds one JSON file listing the logical stations of Île-de-France with their lines, in a strict numbered format (format 1) validated by a closed JSON Schema shipped in the package.

  • Station = IDFM zone de correspondance (multimodal hub), with its id, name and official point converted from Lambert 93 to WGS 84.
  • Lines with IDFM id, label, mode, colours; areas = stop areas served (what SIRI Stop Monitoring queries).
  • Eight modes (METRO, RER, TRAIN, TRAM, BUS, SHUTTLE, FUNICULAR, CABLEWAY); --modes filters. No notion of city or consumer.
  • Header carries version (most recent portal data_processed), generator, license (ODbL-1.0), attribution, modes.
  • Any source inconsistency fails the export instead of dropping data silently.

Why

The Everyday France app embeds this referential offline and owns its contract; this repository is its first producer. Two sources (arrets-lignes, referentiel-des-lignes) are ODbL, so every export is published under ODbL with its notice in the file.

Changes

  • prim_api/referential/: format1.py (pure construction rules), schema.py, cli.py, schemas/stations.format-1.schema.json.
  • manifests/datasets.yml: arrets, zones-de-correspondance (+ generated schemas in data/schema/).
  • prim_api/datasets.py: explicit data_dir, load_metadata, fetch_dataset_metadata, record_metadata_fields (backward compatible).
  • prim_api/__init__.py: IdFMPrimAPI imported lazily — the generated client only exists in a checkout, and eager import made the installed package unusable.
  • pyproject.toml: version 1.0.0, build system (so uvx --from git+…@v1.0.0 export-referential works), pyproj, export-referential entry point, Markdown excluded from ruff format (ruff 0.16 now formats Markdown code blocks and would fail CI on untouched docs).
  • CI: runs the exporter from the built wheel outside the checkout. New release.yml: on a v* tag, checks tag = version, lints, tests, exports all modes, publishes export + schema as a GitHub release.
  • README: export section, stability promise (only the export and its schema), versioning, licence, changes. CLAUDE.md: contract rules. Sample samples/browse_referential.py.

How it's tested

  • 97 new tests (187 total) on small real extracts of the five datasets (tests/fixtures/referential/): every construction rule, schema closure, error cases, CLI with --no-download and mocked downloads. Coverage 98 % on prim_api.referential.
  • Real run: Paris subset (metro, RER, train, tram) = 987 stations, 418 KB; all modes = 15,126 stations, 10 MB; deterministic for identical data; all five datasets valid against their schemas.
  • Wheel checked outside the checkout (uvx --from dist/*.whl): schema packaged, cache dir used, generator = idfm-prim-api@1.0.0. actionlint clean.

To test by hand: uv sync && uv run export-referential --modes METRO,RER,TRAIN,TRAM -o /tmp/stations.json (~40 s, prints the station count per mode).

Limits

  • release.yml has not run yet: it runs on the first tag.
  • The sync tools' console scripts (sync-datasets…) still do not work (tools/ is not packaged) — pre-existing; the README now documents python -m tools.…, as CI already does.
  • The nightly sync has not run since April (GitHub suspends schedules on inactive repositories).

After merge

Push the tag v1.0.0 yourself: it freezes format 1 and triggers the release. Tags are never pushed by an agent.

🤖 Generated with Claude Code

https://claude.ai/code/session_01L4aZBkZFaLEiu7ax7RdwiD

export-referential builds one JSON file of Île-de-France logical stations
(zones de correspondance) with their lines from five open datasets, and
validates it against a closed JSON Schema shipped in the package. A release
workflow publishes the all-modes export and the schema on each tag, under
ODbL. Adds the arrets and zones-de-correspondance datasets, a lazy SDK
import (so the package works outside a checkout), a build system and pyproj.
@MaxLeb

MaxLeb commented Sep 27, 2026

Copy link
Copy Markdown
Owner Author

Agent review

Checked: ruff lint and format, 187 tests (97 new, 98 % coverage on prim_api.referential), actionlint on all workflows, real exports (987 / 15,126 stations), determinism, wheel run outside the checkout.

Worth reading first

  • prim_api/referential/format1.py — every rule there is part of format 1 and is frozen by the v1.0.0 tag.
  • prim_api/__init__.py — the lazy import of IdFMPrimAPI. from prim_api import IdFMPrimAPI keeps working (tested), but anything relying on the generated client being loaded as a side effect of import prim_api would now load it later.
  • pyproject.toml — the new build system makes uv sync install the project in editable mode.

Not tested: release.yml end to end (it only runs on a tag), and gh release create permissions on this repository.

Out of scope, flagged: the broken sync-* console scripts (pre-existing), and the suspended nightly sync.

Not merged, not tagged: both are yours.

The PRIM portal now answers 403 to automated clients, on the API page and on
the direct spec URL alike, so this step failed on main whatever the code. The
sync logic is covered by unit tests; the dry-run stays as a visible warning.
@MaxLeb

MaxLeb commented Sep 27, 2026

Copy link
Copy Markdown
Owner Author

Follow-up: CI initially failed on the network sync dry-run. The PRIM portal now answers 403 to automated clients, on both the API page and the direct spec URL, and the same failure reproduces on main. Commit 5a4561f makes that step non-blocking with a warning. Revert it if you would rather keep it blocking. CI is now green, including the wheel check.

@MaxLeb
MaxLeb merged commit 1570476 into main Sep 27, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant