Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 11 additions & 7 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ name: Release
# The tag is pushed by a human after the PR is merged: publishing a version
# engages the public contract (format + schema), so it is never automated.
# The release carries, under ODbL 1.0 (two IDFM sources are ODbL):
# - stations.format-1.json export of ALL modes (the public offer of
# the derived database)
# - stations.format-1.schema.json the closed JSON Schema of format 1
# - stations.format-<n>.json export of ALL modes, for every produced
# format (the public offer of the derived database)
# - stations.format-<n>.schema.json the closed JSON Schema of each format
# Consumers that embed a subset re-run the exporter with --modes; `generator`
# and `modes` in the file are enough to reproduce it.

Expand Down Expand Up @@ -45,18 +45,22 @@ jobs:
uv run ruff format --check .
uv run pytest

- name: Export the referential (all modes) and copy its schema
- name: Export the referential (all modes, every format) and copy the schemas
run: |
mkdir -p dist
uv run export-referential --format 1 --output dist/stations.format-1.json
cp prim_api/referential/schemas/stations.format-1.schema.json dist/
# The latest format downloads all its datasets (a superset of the older ones); the
# other formats reuse them (--no-download), so every file describes the same data.
uv run export-referential --format 2 --output dist/stations.format-2.json
uv run export-referential --no-download --format 1 --output dist/stations.format-1.json
cp prim_api/referential/schemas/stations.format-*.schema.json dist/

- name: Create the GitHub release
env:
GH_TOKEN: ${{ github.token }}
run: |
cat > notes.md <<'NOTES'
Stations referential of Île-de-France (all modes) and its JSON Schema.
Stations referential of Île-de-France (all modes), in every produced format,
with the JSON Schema of each format.

**Licence**: the referential is a derived database of Île-de-France Mobilités
open data and is published under the [ODbL 1.0](https://opendatacommons.org/licenses/odbl/1-0/).
Expand Down
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,11 @@ tools/ # CLI scripts: sync_specs, generate_clients, sync_datasets,
- **Pipeline order** (`sync_all.py`): sync_specs → generate_clients → sync_datasets → validate_datasets.
- **Python SDK** (`prim_api/`): `IdFMPrimAPI` wraps the generated client with a clean interface, auto-downloads datasets on first use, and refreshes them in a background thread. Core dataset logic lives in `prim_api/datasets.py` (shared with CLI tools).
- **Refs module** (`prim_api/refs.py`): IDFM ↔ STIF identifier conversion helpers (`StopPointRef`, `StopAreaRef`, `LineRef`, `parse_stop_ref`, `parse_line_ref`).
- **Referential exporter** (`prim_api/referential/`): builds the stations referential from five datasets (`zones-de-correspondance`, `zones-d-arrets`, `arrets`, `arrets-lignes`, `referentiel-des-lignes`). `format1.py` is pure (no network, no files) and holds every construction rule; `cli.py` (`export-referential`) downloads, reads `data_processed` from the portal metadata, builds, validates against the schema, writes. `import prim_api` must stay light: `IdFMPrimAPI` is imported lazily because the generated client only exists in a checkout.
- **Referential exporter** (`prim_api/referential/`): format 1 from five datasets, format 2 (format 1 + accesses, `format2.py`) from seven; the CLI keeps a `FORMATS` registry (sources, builder). It builds the stations referential from five datasets (`zones-de-correspondance`, `zones-d-arrets`, `arrets`, `arrets-lignes`, `referentiel-des-lignes`). `format1.py` is pure (no network, no files) and holds every construction rule; `cli.py` (`export-referential`) downloads, reads `data_processed` from the portal metadata, builds, validates against the schema, writes. `import prim_api` must stay light: `IdFMPrimAPI` is imported lazily because the generated client only exists in a checkout.

## Referential contract rules (do not break)

- The format is **strict**: any change of shape (field, mode value, construction rule, sort order) is a **new format number** with its own schema file; never edit `stations.format-1.schema.json` or the format 1 rules once `v1.0.0` is tagged.
- The format is **strict**: any change of shape (field, mode value, construction rule, sort order) is a **new format number** with its own schema file; never edit a released format's schema or rules (format 1 frozen by `v1.0.0`, format 2 by `v1.1.0`). A new format keeps producing the previous ones (minor release).
- The exporter stays **generic**: options are `--format` and `--modes` only; no city, profile or consumer-specific notion.
- Source inconsistencies raise `ReferentialError`; never drop data silently.
- SemVer: new format while keeping the old ones = minor; dropping a format = major. Update the README "Changes" section in every PR that changes behaviour.
Expand Down
53 changes: 44 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,18 +93,18 @@ about cities or consumers: it always covers the whole network, and its only opti

```bash
# From a tag, without cloning (what consumers should pin):
uvx --from git+https://github.com/MaxLeb/idfm-prim-api@v1.0.0 \
export-referential --format 1 --modes METRO,RER,TRAIN,TRAM > stations.json
uvx --from git+https://github.com/MaxLeb/idfm-prim-api@v1.1.0 \
export-referential --format 2 --modes METRO,RER,TRAIN,TRAM > stations.json

# From a checkout:
uv run export-referential --modes METRO,RER -o stations.json
uv run export-referential --no-download -o stations.json # reuse downloaded datasets
# From a checkout (always pass --format: the default stays 1 for compatibility):
uv run export-referential --format 2 --modes METRO,RER -o stations.json
uv run export-referential --format 2 --no-download -o stations.json # reuse downloaded datasets
```

Datasets are downloaded to `data/raw/` in a checkout, or to `~/.cache/idfm-prim-api/raw`
(`$XDG_CACHE_HOME`) when installed. The file is validated against its schema before being written.
Each [release](https://github.com/MaxLeb/idfm-prim-api/releases) also publishes the export of all
modes and the schema.
Each [release](https://github.com/MaxLeb/idfm-prim-api/releases) also publishes, for every produced
format, the export of all modes and its schema.

### Format 1

Expand Down Expand Up @@ -159,16 +159,42 @@ Sources: `zones-de-correspondance`, `zones-d-arrets`, `arrets` (Licence Ouverte
`arrets-lignes`, `referentiel-des-lignes` (ODbL). Any inconsistency in them (unresolvable stop,
line, area or hub, unknown mode) fails the export rather than dropping data silently.

### Format 2

Format 2 is format 1 plus, for every station, the **accesses** (entrances and exits) of its stop
areas, from the IDFM datasets `acces` and `relations-acces` (Licence Ouverte 2.0). Consumers that do
not need them keep asking for `--format 1`, which stays produced.

```json
{ "id": "IDFM:474151", "name": "Châtelet - Les Halles", "…": "…",
"accesses": [
{ "id": "IDFM:50148652", "name": "Porte Marguerite de Navarre", "number": 1,
"lat": 48.860433, "lon": 2.346346, "entry": true, "exit": true }
] }
```

| Field | Meaning |
|---|---|
| `accesses[].id` | `IDFM:<access id>`: the id that platform positioning (`positionnement-dans-la-rame`) points to. |
| `name`, `number` | Name as published; number shown on the signage, or `null`. |
| `lat`, `lon` | Published WGS 84 point, six decimals. |
| `entry`, `exit` | Whether one can enter, exit, or both. |

Rules: every access linked to one of the station's `areas` (the stop areas of the kept modes), each
listed once, sorted by numeric id; `[]` when none is published (tram stops, a few TER stations
outside Île-de-France). The `version` is the most recent `data_processed` of the **seven** datasets.

### Versioning

- **The format is strict**: every field is always present (`null` when the source has no value),
the schema is closed (`prim_api/referential/schemas/stations.format-1.schema.json`), and **any**
each schema is closed (`prim_api/referential/schemas/stations.format-<n>.schema.json`), and **any**
change of shape — a field, a mode value, a construction rule — is a **new format number**.
Consumers read exactly the format they know and refuse the others.
- **The repository follows SemVer on that contract**: producing a new format while still producing
the previous ones is a **minor** release; dropping a format is a **major** release; a fix that
keeps format and output unchanged is a **patch**.
- Format 1 is frozen by `v1.0.0`. Tags are pushed by a maintainer, never by automation.
- Format 1 is frozen by `v1.0.0`, format 2 by `v1.1.0`. Tags are pushed by a maintainer, never by
automation.
- The format is owned by its first consumer (the Everyday France app); this repository is its first
producer and hosts its schema.

Expand All @@ -181,6 +207,15 @@ Releases make the full export available; a filtered export is reproducible from

## Changes

### v1.1.0

- New: **format 2** (`--format 2`, `prim_api.referential.format2`, closed schema
`stations.format-2.schema.json`): format 1 plus the accesses of each station. Format 1 is still
produced unchanged.
- New datasets in `manifests/datasets.yml`: `acces`, `relations-acces`.
- The release publishes every format (all modes) and every schema.
- CLI: one registry of formats (sources and builder); the summary counts accesses for format 2.

### v1.0.0

- New: `export-referential` and `prim_api.referential` — stations referential, **format 1**, closed
Expand Down
98 changes: 98 additions & 0 deletions data/schema/acces.schema.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"accid": {
"type": [
"string",
"null"
]
},
"accversion": {
"type": [
"string",
"null"
]
},
"acccreated": {
"type": [
"string",
"null"
],
"format": "date-time"
},
"accchanged": {
"type": [
"string",
"null"
],
"format": "date-time"
},
"fournisseurid": {
"type": [
"string",
"null"
]
},
"fournisseurname": {
"type": [
"string",
"null"
]
},
"accname": {
"type": [
"string",
"null"
]
},
"accshortname": {
"type": [
"integer",
"null"
]
},
"accdescription": {
"type": [
"string",
"null"
]
},
"accprivatecode": {
"type": [
"string",
"null"
]
},
"accxespg2154": {
"type": [
"integer",
"null"
]
},
"accyespg2154": {
"type": [
"integer",
"null"
]
},
"accisentry": {
"type": [
"string",
"null"
]
},
"accisexit": {
"type": [
"string",
"null"
]
},
"accgeopoint": {
"type": [
"object",
"null"
]
}
}
}
30 changes: 30 additions & 0 deletions data/schema/relations-acces.schema.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"zdaid": {
"type": [
"string",
"null"
]
},
"version": {
"type": [
"string",
"null"
]
},
"accid": {
"type": [
"string",
"null"
]
},
"accversion": {
"type": [
"string",
"null"
]
}
}
}
9 changes: 9 additions & 0 deletions manifests/datasets.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,12 @@ datasets:
portal_base: "https://data.iledefrance-mobilites.fr"
export_format: "jsonl"
validate: true
# Format 2 of the referential: entrances and exits of stations, and their stop areas.
- dataset_id: "acces"
portal_base: "https://data.iledefrance-mobilites.fr"
export_format: "jsonl"
validate: true
- dataset_id: "relations-acces"
portal_base: "https://data.iledefrance-mobilites.fr"
export_format: "jsonl"
validate: true
3 changes: 3 additions & 0 deletions prim_api/referential/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
numbered, strict format validated by a closed JSON Schema.

- ``format1``: construction rules of format 1 (pure, unit-tested);
- ``format2``: format 1 plus the accesses (entrances, exits) of each station;
- ``schema``: the JSON Schemas shipped with the package;
- ``cli``: the ``export-referential`` command.

Expand All @@ -25,6 +26,7 @@
map_mode,
parse_modes,
)
from prim_api.referential.format2 import build_stations_file_format2
from prim_api.referential.schema import load_schema, validate_stations_file

__all__ = [
Expand All @@ -36,6 +38,7 @@
"SOURCE_DATASETS",
"ReferentialError",
"build_stations_file",
"build_stations_file_format2",
"load_schema",
"map_mode",
"parse_modes",
Expand Down
Loading
Loading