Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 21 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,5 +45,24 @@ jobs:
minColorRange: 50
maxColorRange: 100

- name: Dry-run sync pipeline
run: uv run python -m tools.sync_all --dry-run
# Network smoke test only: the sync logic itself is covered by unit tests.
# Since 2026 the PRIM portal answers 403 to automated clients (page and
# direct spec URL alike), which failed this step on main regardless of the
# code. It stays visible (annotation) but no longer blocks a change.
- name: Dry-run sync pipeline (non-blocking)
continue-on-error: true
run: |
uv run python -m tools.sync_all --dry-run || {
echo "::warning::sync dry-run failed (PRIM portal unreachable from CI?)"
exit 1
}

# The exporter is what consumers pin (uvx from a tag): check that it works
# from the built wheel, outside the checkout, with its schema packaged.
- name: Check the installed exporter
run: |
uv build --wheel -o /tmp/dist
cd /tmp
uvx --from /tmp/dist/prim_api-*.whl export-referential --help
uvx --from /tmp/dist/prim_api-*.whl python -c \
"from prim_api.referential import load_schema; load_schema(1)"
69 changes: 69 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
name: Release

# Publishes the stations referential for every version tag (vX.Y.Z).
#
# The tag is pushed by a human after the PR is merged: publishing a version
# engages the public contract (format + schema), so it is never automated.
# The release carries, under ODbL 1.0 (two IDFM sources are ODbL):
# - stations.format-1.json export of ALL modes (the public offer of
# the derived database)
# - stations.format-1.schema.json the closed JSON Schema of format 1
# Consumers that embed a subset re-run the exporter with --modes; `generator`
# and `modes` in the file are enough to reproduce it.

on:
push:
tags: ["v*"]

permissions:
contents: write

jobs:
release:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: astral-sh/setup-uv@v4
with:
version: "latest"

- name: Install dependencies
run: uv sync

- name: Check that the tag matches the package version
run: |
VERSION=$(uv run python -c "import importlib.metadata as m; print(m.version('prim-api'))")
if [ "${GITHUB_REF_NAME}" != "v${VERSION}" ]; then
echo "::error::tag ${GITHUB_REF_NAME} does not match pyproject version ${VERSION}"
exit 1
fi

- name: Lint and test
run: |
uv run ruff check .
uv run ruff format --check .
uv run pytest

- name: Export the referential (all modes) and copy its schema
run: |
mkdir -p dist
uv run export-referential --format 1 --output dist/stations.format-1.json
cp prim_api/referential/schemas/stations.format-1.schema.json dist/

- name: Create the GitHub release
env:
GH_TOKEN: ${{ github.token }}
run: |
cat > notes.md <<'NOTES'
Stations referential of Île-de-France (all modes) and its JSON Schema.

**Licence**: the referential is a derived database of Île-de-France Mobilités
open data and is published under the [ODbL 1.0](https://opendatacommons.org/licenses/odbl/1-0/).
Attribution: « Île-de-France Mobilités ». The code of this repository is MIT.

See the README (« Stations referential export » and « Changes ») for the format,
the stability promise and what changed in this version.
NOTES
gh release create "${GITHUB_REF_NAME}" dist/* \
--title "${GITHUB_REF_NAME}" --notes-file notes.md --verify-tag
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@ build/
# uv
uv.lock

# Coverage reports (pytest-cov)
.coverage
coverage.json

# Dataset artifacts (large, downloaded on demand by devs)
data/raw/
data/reports/
Expand Down
16 changes: 14 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co

## Project Overview

**prim-api** syncs PRIM (Île-de-France Mobilités) OpenAPI/Swagger specs and Opendatasoft datasets, generates Python clients, and validates dataset records against JSON Schema. Everything is driven from YAML manifests and scripts are idempotent.
**prim-api** exports a stations referential of Île-de-France (stable, versioned contract), syncs PRIM (Île-de-France Mobilités) OpenAPI/Swagger specs and Opendatasoft datasets, generates Python clients, and validates dataset records against JSON Schema. Everything is driven from YAML manifests and scripts are idempotent.

## Stack

Expand All @@ -27,7 +27,8 @@ uv run pytest tests/test_foo.py # run a single test file
uv run pytest -k test_name # run a single test by name
uv run ruff check . # lint
uv run ruff format . # format
uv run python tools/sync_all.py # full sync pipeline
uv run python -m tools.sync_all # full sync pipeline (tools/ is not packaged: always `python -m`)
uv run export-referential --modes METRO,RER -o stations.json # referential export
uv run pytest --cov=prim_api # run tests with coverage
uv run pdoc prim_api -o docs/site # generate API docs locally
```
Expand All @@ -36,6 +37,7 @@ uv run pdoc prim_api -o docs/site # generate API docs locally

```
prim_api/ # Python SDK: IdFMPrimAPI wrapper, dataset sync/access, background updater
referential/ # Stations referential exporter: format1.py (pure rules), schema.py, cli.py, schemas/
samples/ # Runnable examples — update when adding new endpoints or datasets
manifests/ # YAML manifests driving all sync (apis.yml, datasets.yml, urls_of_interest.yml)
specs/ # Downloaded OpenAPI/Swagger JSON + .meta.json (committed)
Expand All @@ -56,6 +58,15 @@ tools/ # CLI scripts: sync_specs, generate_clients, sync_datasets,
- **Pipeline order** (`sync_all.py`): sync_specs → generate_clients → sync_datasets → validate_datasets.
- **Python SDK** (`prim_api/`): `IdFMPrimAPI` wraps the generated client with a clean interface, auto-downloads datasets on first use, and refreshes them in a background thread. Core dataset logic lives in `prim_api/datasets.py` (shared with CLI tools).
- **Refs module** (`prim_api/refs.py`): IDFM ↔ STIF identifier conversion helpers (`StopPointRef`, `StopAreaRef`, `LineRef`, `parse_stop_ref`, `parse_line_ref`).
- **Referential exporter** (`prim_api/referential/`): builds the stations referential from five datasets (`zones-de-correspondance`, `zones-d-arrets`, `arrets`, `arrets-lignes`, `referentiel-des-lignes`). `format1.py` is pure (no network, no files) and holds every construction rule; `cli.py` (`export-referential`) downloads, reads `data_processed` from the portal metadata, builds, validates against the schema, writes. `import prim_api` must stay light: `IdFMPrimAPI` is imported lazily because the generated client only exists in a checkout.

## Referential contract rules (do not break)

- The format is **strict**: any change of shape (field, mode value, construction rule, sort order) is a **new format number** with its own schema file; never edit `stations.format-1.schema.json` or the format 1 rules once `v1.0.0` is tagged.
- The exporter stays **generic**: options are `--format` and `--modes` only; no city, profile or consumer-specific notion.
- Source inconsistencies raise `ReferentialError`; never drop data silently.
- SemVer: new format while keeping the old ones = minor; dropping a format = major. Update the README "Changes" section in every PR that changes behaviour.
- The version tag (`vX.Y.Z`, must equal `pyproject.toml`) is pushed by the maintainer, never by an agent; the `release.yml` workflow then publishes the all-modes export and the schema under ODbL.

## Environment Variables

Expand All @@ -66,3 +77,4 @@ tools/ # CLI scripts: sync_specs, generate_clients, sync_datasets,
- **ci.yml** — on PR/push: install, test (with coverage), lint, dry-run sync. Pushes coverage badge to gist on main.
- **nightly-sync.yml** — nightly at 02:00 Europe/Paris: sync specs + regenerate clients, auto-open PR if changes detected. Dataset sync is not part of nightly (devs download on demand).
- **docs.yml** — on push to main: builds API docs with pdoc and deploys to GitHub Pages.
- **release.yml** — on version tag: checks tag = package version, lints, tests, exports the referential (all modes), publishes it with its schema as a GitHub release.
139 changes: 129 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,17 @@
[![Docs](https://img.shields.io/badge/docs-GitHub%20Pages-blue)](https://MaxLeb.github.io/idfm-prim-api/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)

> **Work in progress** — This project is an experiment in "vibe coding" with AI assistance. It comes with no guarantee of correctness, completeness, or stability. Use at your own risk.
> **Work in progress** — This project is an experiment in "vibe coding" with AI assistance. It comes with no guarantee of correctness or completeness. Use at your own risk.
>
> **Stability promise (since v1.0.0)** — only the [stations referential export](#stations-referential-export) is stable: its format and its JSON Schema follow SemVer (see [Versioning](#versioning)). Everything else (SDK, generated clients, sync tools) stays experimental and may change in any release.

Auto-sync PRIM (Île-de-France Mobilités) OpenAPI/Swagger specs, generate Python clients, and sync/validate Opendatasoft datasets.
Export a stations referential of Île-de-France, auto-sync PRIM (Île-de-France Mobilités) OpenAPI/Swagger specs, generate Python clients, and sync/validate Opendatasoft datasets.

This repository maintains up-to-date interface contracts from PRIM APIs and dataset exports from the IDFM Opendatasoft portal. Everything is manifest-driven and idempotent.

## What it does

- **Exports a stations referential** of Île-de-France (logical stations with their lines, one JSON file, strict numbered format, closed JSON Schema) — see [Stations referential export](#stations-referential-export)
- **Syncs OpenAPI/Swagger specs** from PRIM APIs (supports direct URLs and PRIM page scraping)
- **Generates Python clients** from specs using OpenAPI Generator
- **Downloads dataset exports** from Opendatasoft (JSONL format, full exports without pagination limits)
Expand All @@ -36,6 +39,7 @@ Each step is conditional: resources are only re-fetched or regenerated when chan

```
prim_api/ # Python SDK (IdFMPrimAPI, dataset sync, background updater)
referential/ # Stations referential exporter + JSON Schemas (stable contract)
samples/ # Runnable usage examples (update when adding endpoints/data)
manifests/ # YAML manifests (apis.yml, datasets.yml, urls_of_interest.yml)
specs/ # Downloaded OpenAPI/Swagger specs (committed)
Expand All @@ -59,6 +63,8 @@ docs/site/ # Generated API docs (gitignored)
- OpenAPI specs + metadata (`specs/`) — updated by nightly sync
- Generated Python clients (`generated/clients/`) — regenerated when specs change
- Dataset schemas (`data/schema/`) — kept in sync with portal metadata
- Referential schemas (`prim_api/referential/schemas/`) — the contract, changed only with a new format
- Test fixtures (`tests/fixtures/referential/`) — small real extracts of the five source datasets

**Gitignored (downloaded on demand by devs):**
- `data/raw/` — dataset exports (JSONL, can be large)
Expand All @@ -78,6 +84,115 @@ docs/site/ # Generated API docs (gitignored)
uv sync
```

## Stations referential export

`export-referential` builds, from five IDFM open datasets, a single JSON file listing the **logical
stations** of Île-de-France with their lines — meant to be embedded offline in apps. It knows nothing
about cities or consumers: it always covers the whole network, and its only options are the
**format** and the **modes** to keep.

```bash
# From a tag, without cloning (what consumers should pin):
uvx --from git+https://github.com/MaxLeb/idfm-prim-api@v1.0.0 \
export-referential --format 1 --modes METRO,RER,TRAIN,TRAM > stations.json

# From a checkout:
uv run export-referential --modes METRO,RER -o stations.json
uv run export-referential --no-download -o stations.json # reuse downloaded datasets
```

Datasets are downloaded to `data/raw/` in a checkout, or to `~/.cache/idfm-prim-api/raw`
(`$XDG_CACHE_HOME`) when installed. The file is validated against its schema before being written.
Each [release](https://github.com/MaxLeb/idfm-prim-api/releases) also publishes the export of all
modes and the schema.

### Format 1

```json
{
"format": 1,
"version": "2026-09-27T07:05:39Z",
"generatedAt": "2026-09-27T09:00:00Z",
"generator": "idfm-prim-api@1.0.0",
"source": "idfm-opendata",
"license": "ODbL-1.0",
"attribution": "Île-de-France Mobilités",
"modes": ["METRO", "RER", "TRAIN", "TRAM"],
"stops": [
{ "id": "IDFM:474151", "name": "Châtelet - Les Halles", "lat": 48.86174, "lon": 2.34697,
"areas": ["IDFM:monomodalStopPlace:45102"],
"lines": [
{ "id": "IDFM:C01742", "label": "A", "mode": "RER", "color": "EB2132", "textColor": "FFFFFF" }
] }
]
}
```

| Field | Meaning |
|---|---|
| `format` | Version of the data contract (integer). |
| `version` | Most recent `data_processed` date among the five source datasets (UTC): changes if and only if the content may have changed. |
| `generatedAt` | Time of the export (UTC). |
| `generator` | `idfm-prim-api@<version>` that produced the file. With `modes`, enough to reproduce it. |
| `source`, `license`, `attribution` | `idfm-opendata`, `ODbL-1.0`, « Île-de-France Mobilités ». |
| `modes` | Modes kept at export time: a station of another mode was not exported, not absent. |
| `stops[].id`, `name`, `lat`, `lon` | The IDFM **zone de correspondance** (multimodal hub): its id, its name, its official point converted from Lambert 93 to WGS 84 (six decimals). |
| `stops[].areas` | Stop areas of the hub served by a kept line (what SIRI Stop Monitoring queries). |
| `stops[].lines[]` | IDFM line id, label, mode, colours (six uppercase hex digits, or `null`). |

Construction rules (all part of the format): a station is a **zone de correspondance**, never a
grouping computed by distance or name; only `active` lines are kept; a hub without a kept line is
not exported; stations are sorted by id, lines by mode then natural label order then id. Modes:

| Value | IDFM modes |
|---|---|
| `METRO` | metro |
| `RER` | RER A to E (`rail` / `local`) |
| `TRAIN` | Transilien H to V, TER |
| `TRAM` | tramway |
| `BUS` | bus (all sub-modes) |
| `SHUTTLE` | CDG VAL, Orlyval |
| `FUNICULAR` | Montmartre funicular |
| `CABLEWAY` | Câble C1 |

Sources: `zones-de-correspondance`, `zones-d-arrets`, `arrets` (Licence Ouverte 2.0) and
`arrets-lignes`, `referentiel-des-lignes` (ODbL). Any inconsistency in them (unresolvable stop,
line, area or hub, unknown mode) fails the export rather than dropping data silently.

### Versioning

- **The format is strict**: every field is always present (`null` when the source has no value),
the schema is closed (`prim_api/referential/schemas/stations.format-1.schema.json`), and **any**
change of shape — a field, a mode value, a construction rule — is a **new format number**.
Consumers read exactly the format they know and refuse the others.
- **The repository follows SemVer on that contract**: producing a new format while still producing
the previous ones is a **minor** release; dropping a format is a **major** release; a fix that
keeps format and output unchanged is a **patch**.
- Format 1 is frozen by `v1.0.0`. Tags are pushed by a maintainer, never by automation.
- The format is owned by its first consumer (the Everyday France app); this repository is its first
producer and hosts its schema.

### Licence of the export

The referential is a derived database of Île-de-France Mobilités open data, two of its sources being
under ODbL: **every export is published under the ODbL 1.0**, with the attribution
« Île-de-France Mobilités », carried by the `license` and `attribution` fields of the file itself.
Releases make the full export available; a filtered export is reproducible from `generator` + `modes`.

## Changes

### v1.0.0

- New: `export-referential` and `prim_api.referential` — stations referential, **format 1**, closed
JSON Schema, release workflow publishing the export (all modes) and the schema.
- New datasets in `manifests/datasets.yml`: `arrets`, `zones-de-correspondance`.
- `prim_api.datasets`: explicit `data_dir`, `load_metadata`, `fetch_dataset_metadata` (portal
`data_processed`, the export API gives no ETag/Last-Modified), `record_metadata_fields`.
- `import prim_api` no longer loads the generated PRIM client eagerly (`IdFMPrimAPI` is imported on
first access), so the package works when installed outside a checkout.
- Packaging: build system declared (`uvx --from git+…@<tag>` works), `pyproj` dependency, stability
promise limited to the export.

## Python SDK

The `prim_api` package provides a high-level Python interface to PRIM APIs and datasets.
Expand Down Expand Up @@ -160,34 +275,37 @@ See [`samples/`](samples/) for runnable examples.

## CLI Tools

The sync tools live in `tools/`, which is not part of the installed package: run them as modules
from a checkout.

### Run individual steps

```bash
# Sync OpenAPI/Swagger specs
uv run sync-specs
uv run python -m tools.sync_specs

# Generate Python clients
uv run generate-clients
uv run python -m tools.generate_clients

# Download datasets
uv run sync-datasets
uv run python -m tools.sync_datasets

# Validate datasets
uv run validate-datasets
uv run python -m tools.validate_datasets
```

### Run the full pipeline

```bash
uv run sync-all
uv run python -m tools.sync_all
```

### Dry-run mode

Most tools support `--dry-run` to preview changes without modifying files:

```bash
uv run sync-all --dry-run
uv run python -m tools.sync_all --dry-run
```

### Development commands
Expand Down Expand Up @@ -228,7 +346,7 @@ Runs nightly at 01:00 UTC (≈ 02:00 Europe/Paris):
2. Regenerates Python clients if specs changed
3. Opens a PR automatically if anything changed

Dataset sync is **not** part of the nightly — devs download data locally on demand via `uv run sync-datasets`.
Dataset sync is **not** part of the nightly — devs download data locally on demand via `uv run python -m tools.sync_datasets`.

### `docs.yml` (API docs)

Expand Down Expand Up @@ -293,5 +411,6 @@ API reference is auto-generated from docstrings and published to GitHub Pages:

This project's own code is licensed under the [MIT License](LICENSE).

- **IDFM data** — datasets and API responses from Île-de-France Mobilités are published under the [Open Database License (ODbL 1.0)](https://spdx.org/licenses/ODbL-1.0.html).
- **IDFM data** — datasets and API responses from Île-de-France Mobilités are published under the [Open Database License (ODbL 1.0)](https://spdx.org/licenses/ODbL-1.0.html) or the Licence Ouverte 2.0 (Etalab), depending on the dataset.
- **Stations referential export** — a derived database, published under the ODbL 1.0 with the attribution « Île-de-France Mobilités » (see [Licence of the export](#licence-of-the-export)). Test fixtures in `tests/fixtures/referential/` are small extracts of the source datasets, under their own licences.
- **Generated clients** — Python clients in `generated/clients/` are produced by [OpenAPI Generator](https://openapi-generator.tech/), licensed under [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0).
Loading
Loading