Repository navigation
Conversation
_resolve_map_file reads PY123D_DATA_ROOT from the environment rather than taking the root it was handed, so any dataset whose map is shared across logs (map_is_per_log=False: nuplan, nuscenes, nurec) fails with "Map API is required to convert scenario" unless the caller also exports that variable. Set it from --py123d_path so the documented CLI flag is enough on its own.
process_traffic_controls skipped every TRAFFIC_LIGHT stop zone whenever the scenario already carried traffic lights, on the reasoning that a signalised junction should not emit a duplicate control. That deduplication is already done, per lane, by the covered_lanes filter two lines below, and the all-or-nothing guard defeats it: a stop zone spanning several lanes is dropped entirely even when only one of them has a light. test_static_traffic_light_keeps_only_uncovered_lanes asserts exactly that — a dynamic light on lane 1 and a stop zone controlling lanes 1 and 2 must yield two controls, [[1], [2]]. The test fails on this commit's parent and passes here; the suite goes from 51 failures to 50 with none introduced. The case that surfaced it: a dataset can record stop-line geometry and no light states, and then the guard drops the junction's only usable control. On a 100-scene NuRec set, yield controls fell to 23 of 177 against 73 without it.
GENERIC_OBJECT was mapped into the objects section, which the PufferDrive
loader parses but never reads: nothing collides with it, and no observation
includes it. Anything the source data could not classify therefore vanished
from the simulation even though it physically blocks the road.
This is not specific to one dataset — 6 of the 11 py123d taxonomies emit the
label (nuPlan, nuScenes, KITTI-360, PandaSet, nuReasoning, PhysicalAI-AV) and
it is invisible in all of them, so it belongs in the shared map rather than
behind a per-dataset flag.
Measured on 11 NuRec scenarios converted from the same Arrow store: before, 4
tracks landed in the objects section and the agent counts were {vehicle 441,
pedestrian 53, cyclist 4}; after, the same 4 appear as AgentType.OTHER agents
and the objects section is empty.
BARRIER and TRAFFIC_CONE are still routed to objects and remain invisible for
the same reason; other datasets may need the same treatment.
Bins are recentred by the ego-trajectory centroid, and the positional bin format has nowhere to carry that offset, so a consumer moving states between PufferDrive and another simulator cannot get back to source coordinates. --emit_metadata (off by default, on in the nurec preset) writes one metadata.jsonl in the output directory, a row per converted scenario holding the centroid plus the bin filename, scenario id, dataset, length and dt. Workers return the row and the parent streams it, the same way failures.jsonl is already produced. Provenance only; the runtime never reads it. Also warn once per scenario about detection labels that neither mapping covers, so a silently dropped agent class shows up in the logs instead of as a missing obstacle.
Pins the scenario id field and turns on --emit_metadata. This is the only dataset-specific change in the branch; everything else applies to any py123d dataset. Deliberately omits reverse_road_edges, unlike the nuplan, nuscenes, carla and opendrive presets: NuRec boundaries already come with the drivable side on the left. Measured against 8 CARLA reference bins (drivable side left in 92-100% of edges), NuRec's own winding is already left in 97%, so setting the flag would mirror every boundary the policy observes.
The exact pin makes 123Drive unusable alongside any project that tracks a different py123d. A workspace holding both fails to resolve at all: Because 123drive==0.3.2 depends on py123d==0.6.0 and there is no version of py123d==0.6.0, we can conclude that all versions of 123drive cannot be used. 123Drive reads the Arrow store through py123d's API rather than depending on a particular writer: converting one fixed Arrow store against py123d 0.6.0 and 0.7.0 gives byte-identical bins across a 97-scenario set. Widen to >=0.6.0,<0.8 — both tested versions, and no claim about 0.8, which nobody has run. That keeps existing 0.6.0 callers working while letting a caller pin 0.7.0 themselves, instead of every such caller carrying a dependency override.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I converted 97 NuRec scenes with stock
py123d==0.6.0and an unmodified 123Driveand got byte-identical bins, so nothing NuRec-specific is needed here. This is 3 fixes
and a preset. Two of the fixes change the resulting binaries for all supported datasets.
traffic_controls.pyThe
TRAFFIC_LIGHT+scenario_length > 0check skips the whole stop zone, so a zonecovering lanes 1 and 2 gets dropped when only lane 1 has a light. The
covered_lanesfilter a few lines down already handles that case per lane.
test_static_traffic_light_keeps_only_uncovered_lanescovers this and fails ondev_v0.3.3. It passes with the check removed. Suite goes from 51 failures to 50.For NuRec this mattered because we have stop lines but no light states, so those
junctions ended up with no control at all. Yield controls went from 23 to 73 out of 177.
mapping.pyGENERIC_OBJECTmaps to the objects section, which the loader seeks past, so thoseboxes aren't collidable. 6 of the 11 py123d taxonomies emit the label, so I put it in
AGENT_TYPE_MAPinstead of adding a flag. On 11/100 NuRec scenarios, 4 tracks move from objectsto agents and nothing else changes.
BARRIERandTRAFFIC_CONEare in the same position. I left them, since I couldn't testthe datasets that emit them.
Rest
_resolve_map_filereadsPY123D_DATA_ROOTfrom the environment rather than the rootpassed in, so
--py123d_pathon its own fails formap_is_per_log=Falsedatasets(nuplan, nuscenes) with "Map API is required to convert scenario". I set the env var
in
discover_scenes.--emit_metadata, off by default, writesmetadata.jsonlwith the centroid from re-centering.The bin format has nowhere to put it and we need it to move states between
PufferDrive and AlpaSim. Written from the workers like
failures.jsonl.