Repository navigation
feat(reading): convert data whose byte order is not the host's - #189
Merged
Merged
Conversation
Apollo3zehn
reviewed
Oct 21, 2026
Big-endian data was refused outright - "Byte order conversion is not (yet) support by PureHDF" - which put a large class of files out of reach, big-endian being HDF5's portable choice and what a good deal of instrument software produces. The refusal was also incomplete in a way that mattered more than the refusal itself. It inspected only the top-level datatype, so anything nested went unguarded: a compound with big-endian members was byte-copied straight into the caller's struct, and (1, 2.0) came back as (16777216, 3.16E-322) with no error at all. The same held for an enumeration's base type, an array datatype's base type, and a variable-length sequence's base type. Byte order belongs to each leaf of a datatype rather than to the dataset, which is why one answer per read could not have been right: a compound can hold one member of each order. Conversion is therefore driven by the datatype tree. The bulk decoders take a layout plan of (offset, width) pairs and reverse those spans in the buffer the data was read into; the element-at-a-time decoders convert per leaf, each member answering for itself. Data already in the host's byte order keeps the existing decoder exactly, capturing nothing and making no extra pass, so the common path is unchanged. VAX floating-point is still refused, being a different representation rather than a permutation of the bytes, and so is a variable-length datatype whose values would need conversion in a place this cannot reach. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…type The per-element unmanaged decoder cache is process-wide and was keyed on the .NET type alone. Converting made that decoder depend on the datatype it was built from, so the first big-endian value of a given type handed its converting decoder to every little-endian value of that type afterwards, for the life of the process - a little-endian 1 read back as 16777216. It surfaced only with an unlucky test order, which is the worst form of it: the cached decoder outlives the file that produced it, so in a process reading several files the corruption depends on which one was opened first. The new EndiannessTests build their fixtures through the HDF5 C library, so they join the collection that serialises access to its process-global state. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… byte orders The cache defect the previous commit fixed was caught only by an unlucky test order, and nothing here would have caught it again: the mixed-endianness dataset was read into a struct, which converts by layout through the bulk path and never consults the per-element cache. Reading the same dataset into a class does consult it, and both members are `int` - so a cache keyed on the .NET type alone has one entry to serve two orders, and whichever member decodes first decides for the other. Reverting the key makes this fail with 117440512 for 7, rather than depending on which file a process opened first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The decoders are cached beyond the lifetime of the file that produced them, so reading either file correctly on its own says nothing about reading both. This opens one big-endian and one little-endian file holding identical structure, in both orders, revisiting each after the other has been read. Every shape is read as the same .NET type in both files, and the shapes are chosen to reach each decoding path that converts: a bulk array, a compound as a packed struct, the same compound as a class, a scalar dataspace, and a variable-length sequence whose values come off the global heap. Disabling either the layout plan or the cache key makes it fail, so it covers both rather than standing in for one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Apollo3zehn
force-pushed
the
feat/big-endian-reads
branch
from
October 22, 2026 10:19
e107e5e to
03880c4
Compare
added 7 commits
October 22, 2026 12:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Big-endian data was refused outright —
"Byte order conversion is not (yet) support by PureHDF"— which puts a large class of files out of reach. Big-endian is HDF5's portable choice and what a good deal of instrument software writes.The refusal was incomplete, which mattered more than the refusal
It inspected only the top-level datatype, so anything nested went unguarded and was byte-copied through:
No exception. The same held for an enumeration's base type, an array datatype's base type, and a variable-length sequence's base type. The honest refusal was the good case; everything it did not reach returned plausible-looking nonsense.
Byte order belongs to the leaf, not the dataset
A compound can hold one member of each order, so no single answer per read can be correct —
mixed_compoundin the fixture is exactly that, and would defeat any per-dataset approach.Conversion is therefore driven by the datatype tree:
(offset, width)pairs collected from the tree, and reverse those spans in the buffer the data was read intoBoth routes are covered by tests, because a struct and a class decode by different paths and only one of them would otherwise be exercised.
The common path is untouched
Data already in the host's byte order gets the existing decoder exactly — the same static delegate, capturing nothing, with no extra pass over the buffer. The conversion is a second pass only where one is needed, and the read itself stays zero-copy either way.
Still refused
VAX floating-point, which is a different representation rather than a permutation of the bytes, and a variable-length datatype whose values would need conversion on the global heap, which this pass cannot reach. Both now fail with a message saying which.
Fixtures
AddEndiannesswrites through the C library with the file type as the memory type, so the bytes are exactly what a big-endian producer would emit rather than something converted on the way in. It covers a big-endian integer and float, a little-endian control, a wholly big-endian compound, a mixed-endianness compound, an enumeration over a big-endian base, and an array member over one.🤖 Generated with Claude Code