feat(lerobot_v3): add image-mode output to the v3 writer - #6
Merged
Conversation
Add `LeRobotV3WriterConfig.video` (default True). With `video=False` the writer PNG-encodes each camera frame into the data parquet as a `struct<bytes, path>` cell and declares the feature as `dtype: "image"` in info.json, matching the LeRobot v3 image-dataset layout and upstream LeRobot's own `video=False` convention. The video encoder is bypassed entirely in that mode and no videos/ directory is created; `video_path` and `video_files_size_in_mb` are null while the keys are retained. Video mode remains the default and its output is unchanged. Also in this change: - carry `writer_config` through `ConversionConfig.to_dict()` and apply it in the parallel conversion worker, so `--workers > 1` honours image mode; - read inline image cells back in the v3 reader, tolerating null and path-only cells; - keep late-appearing parquet columns by filling the union of row keys before building a chunk table; - skip null rows when computing episode stats and when inferring features; - bound finalize's feature sampling to one batch instead of reading whole PNG payloads; - share one `_configure_writer` helper between the sequential and parallel converter paths. Tests cover image-mode dataset structure, a read-back through the LeRobot v3 reader, and a video-mode regression class.
griswaldbrooks
force-pushed
the
fm/forge-image-mode
branch
from
August 21, 2026 23:59
6f9f14c to
e0f78ba
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an image-mode output option to the LeRobot v3 writer: camera frames can be stored as
PNG images inline in the data parquet instead of being encoded to MP4. Video mode remains
the default and its output is unchanged.
What changed
Writer (
forge/formats/lerobot_v3/writer.py) — newLeRobotV3WriterConfig.videoflag(default
True), following upstream LeRobot's ownvideo=Falseconvention. Withvideo=False:struct<bytes, path>column,matching the HF
datasets.Image()storage LeRobot uses for image datasets;info.jsonasdtype: "image";videos/directory is created;video_pathandvideo_files_size_in_mbarenull, with the keys retained because v3loaders require them to exist.
Pillow is required for image mode (added to the
[lerobot]extra) and checked up front.Reader (
forge/formats/lerobot_v3/reader.py) — camera features withdtype: "image"aredecoded from the inline PNG bytes (or a referenced path) through a lazy PIL loader, so
image-mode datasets round-trip through Forge.
Converter (
forge/convert/converter.py,forge/config/models.py) —writer_configiscarried through
ConversionConfig.to_dict()and applied in the parallel worker, so--workers > 1honours image mode. The sequential and parallel paths now share one_configure_writerhelper.Usage:
Tests
tests/test_lerobot_v3_writer.py— image-mode structure (info.json dtype, novideos/,frame counts, null
video_path), plus a video-mode-unchanged regression class.tests/test_lerobot_v3_roundtrip.py— image-mode frames survive write → read through thereal v3 reader.
tests/test_converter.py—writer_configround-trips throughto_dict/from_dict, andthe parallel worker actually applies
video=False.tests/test_lerobot_v3_upstream_validity.py— extended with awritten_v3_imagefixture anda
TestUpstreamLoaderImageModeclass, gated on lerobot being installed like the existingupstream-validity tests.
Verification against stock LeRobot
lerobot is not a test dependency, so the spec check was run manually. In a scratch venv with
lerobot 0.4.4, the real 206-episode pusht zarr sample was converted through
forge convert --configin image mode and loaded with stockLeRobotDataset:image_keysis populated andvideo_keysis empty. The gated upstream-validity suite passes23/23 against real lerobot. Frames decoded by Forge's reader, by stock HuggingFace
datasetsvia
Image(), and byLeRobotDatasetare all pixel-identical to the source zarr.Video mode was compared base vs branch on the same sample: every parquet and JSON artifact is
byte-identical. The MP4 files differ byte-wise, but decoded pixels match, and two runs of the
same build differ by the same margin — x264 run-to-run nondeterminism, not a behaviour change.
Known limitations
Filed as follow-ups, all pre-existing or narrow enough to ride a patch release:
chunk-000(pre-existing; affects videomode equally).
finalize()can misdeclare a late-appearing image column.info.jsonstill declares it.