Repository navigation
Cpp data loader - #126
Merged
Merged
Cpp data loader#126
Conversation
Reading a folder of images went through torchvision::base_loader(), which decodes into an R double array: 24 MB for one 1024x1024 RGB image against 3 MB as uint8. That costs a constant factor of about seven regardless of image size. There was also no parallelism across images, and torch's parallel dataloader cannot supply it: it has no prefetch overlap with backward() and carries roughly two seconds of fixed per-batch overhead, which shrinking the payload fourfold did not recover. src/decode.cpp decodes a whole batch into one pre-allocated buffer at the file's native bit depth and hands that to torch_tensor_from_buffer(), so no R double array is built. Dispatch is on magic bytes rather than on the file extension. Measured against the previous path on identical files: JPEG 1024x1024 19.8 -> 183.8 images/s JPEG 224x224 533.3 -> 5309.7 images/s TIFF 1024x1024, 10 bands, 16-bit 2.2 -> 30.3 images/s Output is bit-exact against the reference R readers for JPEG grey and RGB, PNG grey, RGB and RGBA, TIFF 8- and 16-bit, and 12-band 16-bit both stripped and tiled with DEFLATE. Errors are collected per file and raised in R naming the offending path; nothing inside the parallel region touches the R API. libjpeg gets a per-call setjmp error manager instead of its default exit(), libpng a handler that records the message before the longjmp, and libtiff's process-global handlers are restored on return so that tiff and terra keep theirs. configure probes each library with a compile-and-link test and defines HAVE_JPEG, HAVE_PNG and HAVE_TIFF independently, so a missing -dev package degrades that one format to the R reader instead of failing the install. Multi-page TIFFs are handed back to the R reader, which stacks their directories.
…ves in Three review rounds and an adversarial test pass found defects the first implementation could not have survived in the field. The worst were silent: A float32 batch aliased freed memory. torch_tensor_from_buffer() borrows the raw vector and $to(dtype) copies only when the dtype actually changes, so for a float32 file it was a no-op and the tensor kept pointing at memory R had collected. On a 128-band tile two reads of the same file gave sums of 5.17e20 and 1.63e4. Fixed with an explicit clone. Signed and unsigned 16-bit samples in one batch were decoded under one interpretation, rescaling half of them. A signed elevation model beside an unsigned reflectance band is an ordinary folder, and which half was wrong depended on the shuffle, so the same file differed between epochs. Cloud-optimised GeoTIFFs could not be read at all, by either path. A COG stores overviews, and usually an internal mask, as further directories; the compiled path counted them as pages and refused the file, while load_tiff() passed every directory to readTIFF and tried to stack images of different sizes and bit depths. Only full-resolution directories are used now, identified by SUBFILETYPE. Greyscale PNG with a tRNS chunk returned four channels against the R reader's two, because the gray-to-rgb transform tested the colour type from before tRNS had turned it into grey+alpha. The configure link probe could not fail on a static archive: its main() referenced no symbol, so the linker pulled nothing from libtiff.a and never saw its unresolved dependencies. On CRAN's macOS builders that would have produced a shared object that links and then fails at dlopen, taking the whole package with it. Each probe now calls a symbol. Formats added, because roughly half of the surveyed remote-sensing product families are not 8- or 16-bit unsigned: 32-bit float, which is what per-pixel embedding products are distributed as, and signed 16-bit, used by elevation models; both pass through unscaled, since only unsigned integer samples carry an implied [0, 1] range. Palette images expand through their colormap, and white-is-zero returns the stored samples, which is what GDAL and readTIFF both do. Layouts the compiled path cannot read safely now hand the batch to the R reader rather than guessing: subsampled YCbCr, whose scanlines are shorter than width times samples and once caused a heap over-read, multi-page files, tiled separate planes, and anything that is not 8, 16 or 32 bit. Where the R reader is itself wrong -- band-interleaved planes, signed samples, YCbCr -- it refuses with an explanation instead of returning quietly corrupted pixels. Both paths now dispatch on magic bytes. Deciding by extension made a file without one readable on one path and not the other, which put readability at the mercy of which system library happened to be installed. tests/testthat/test-decode.R covers all of this, including fixtures in the layouts the data actually arrives in: a COG with overviews, an internal mask IFD, float32, signed elevation, band-interleaved, multi-page and subsampled YCbCr. CI: pak::local_system_requirements() has been removed from pak, which is why both Linux jobs failed. The hand-rolled dependency steps are replaced by setup-r-dependencies, and libjpeg, libpng and libtiff are installed explicitly so that the compiled decoder is what gets checked.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.