[Blackclaws]

A fixed-length string attribute's width came from H5WriteOptions.DefaultStringLength, which is file-global: it has to be as wide as the widest attribute anywhere in the file, and every other attribute is then padded out to it. On a real 611 MB report that padding was over half of the file's 42 MB of metadata.

It is also the only defence against data loss, because the encoder silently truncates anything wider — by byte, so a multi-byte character can be cut in half.

A width measured from the value cannot truncate it, and pads only out to the attribute's own longest value rather than to the file's widest — nothing at all for a scalar. So that is what a string attribute now gets.

The width is the attribute's own business

DefaultStringLength was doing two unrelated jobs for attributes: switching them to fixed-length, and then being ignored as the width. That left combinations unreachable — variable-length datasets alongside measured attributes, or a declared compound width alongside variable-length attributes. H5WriteOptions.AttributeStringLength now answers for attributes on its own:

	DefaultStringLength = 0	DefaultStringLength = 32
Measured (default)	fixed-length, measured	fixed-length, measured
Inherit	variable-length	fixed-length at 32, truncates
VariableLength	variable-length	variable-length

file.Write(stream, new H5WriteOptions(DefaultStringLength: 32)
{
    // Attributes take the declared width too - the behaviour before this PR.
    AttributeStringLength = H5AttributeStringLength.Inherit
});

Datasets are deliberately unchanged

A compound member's width has to cover every row and stay uniform across the tables sharing that type, so it stays caller-declared — and that includes the string members of a compound attribute, whatever AttributeStringLength says. For those, the new H5WriteOptions.StringOverflow opts into failing the write instead of truncating:

file.Write(stream, new H5WriteOptions(DefaultStringLength: 32)
{
    StringOverflow = H5StringOverflow.Throw
});

Truncate remains the default there, so nothing changes for declared widths unless asked.

Encoding also now rethrows unwrapped rather than as TargetInvocationException, so a rejected value reports why instead of "Exception has been thrown by the target of an invocation".

Nulls

A fixed-length field has no way to hold the difference between null and "", so measuring steps back rather than losing it: an attribute holding a null element stays variable-length. That is what keeps the argument for measuring intact — it never loses a value.

At a declared width there is nothing to step back to, and a null is already a crash on dev: Encoding.UTF8.GetBytes((string)source) throws ArgumentNullException, for a dataset, a compound member and an attribute alike. Nothing caught it because the fixed-length test data has "" but no null. It is now refused with a message that says what to do instead.

That is the one place this PR reaches datasets — a crash becoming a clear error rather than a change of behaviour, but worth calling out against the section above.

Worth flagging

• This changes default behaviour for attributes. DefaultStringLength no longer governs their width at all, so files written with default options carry fixed-length attribute datatypes where they previously carried variable-length ones. AttributeStringLength.Inherit restores the old behaviour exactly.

• ThePaddingWrittenToTheFileIsZero from fix(writing): zero the padding of a fixed-length string #177 now asserts through a compound member rather than a string attribute. A measured attribute has no padding to inspect, and the compound path is where a width is still declared — so the byte-level assertion still covers the pooled-buffer branch that test was written for.

• A measured width declares H5T_STR_NULLPAD, not NULLTERM. NULLTERM reserves the field's final byte, and the C library honours that when converting a value in, so a tool rewriting the attribute through a wider datatype lands width - 1 bytes. A measured width is filled to its last byte by its longest element by construction, so NULLTERM would put exactly that element at risk. Measured against the C library on a PureHDF-written attribute:

  	as written	after an in-place rewrite through a wider fixed type
  NULLTERM 7	b'Wafer\x00\x00', b'Reticle'	b'Wafer\x00\x00', b'Reticl\x00'
  NULLPAD 7	b'Wafer\x00\x00', b'Reticle'	b'Wafer\x00\x00', b'Reticle'

  The declaration does not affect reads. There is no conversion path between fixed- and variable-length strings in the C library at all, so every consumer reads the field into a fixed-width buffer, and those bytes are identical under both declarations — H5Aread through the file's own type gives b'Wafer\x00\x00' with strlen 5 either way, and h5py and PureHDF both return Wafer.

  The cost is that h5dump honours NULLPAD literally and prints a shorter value with its padding attached, so the attribute dump fixtures show "ABCDEF\000\000\000\000" where they showed "ABCDEF". That is a rendering difference in one tool, not something a reader sees — and the alternative was silently truncating the longest value of every mixed-length attribute.

  Compound members are unaffected: they pass no measured width and keep NULLTERM.

• The datatype cache key gains the padding. It is keyed by everything the message depends on (fix(writing): key the datatype cache by everything the message depends on #173), and an attribute measured at six bytes now differs from a compound member declared at six bytes. Without the extra key component the member silently inherits whichever padding was encoded first; there is a test that fails if it is removed.

• The two global-heap tests in MiscTests now ask for variable-length attributes explicitly. They filled a collection by writing string attributes under default options, which measuring empties out — AttributeStringLength.VariableLength is what those tests are actually about.

• AttributeStringLengthTests covers the three modes, the compound-member carve-out and the null fallback; AttributeStringWidthTests covers the measured widths, the padding and overflow. CanWrite_Common prefers an attr_ dump fixture for the cases where an attribute genuinely differs from the dataset, and shares the dataset fixtures otherwise.

• doc/writing/data-types.md gains a Strings section — DefaultStringLength and StringOverflow were not documented at all.

• Overlaps feat(writing): keep file structure together, opt-in per placement #180 only in H5WriteOptions, where both add a property to the record body; whichever lands second takes a one-line conflict.

[Blackclaws]

The build failure is by the way exactly the bug fixed in the other PR :)

[Apollo3zehn]

Thanks for this PR. Another nice improvement :-) I have applied minor changes, please review commit 0334b9e and tell me if you have doubts. Please see the commit message for the reasoning for the changes.

Apart from that I have trouble reviewing the changes because the changed files contain many PR-unrelated changes like white space removals, var -> named type changes or the change from named parameter to positional parameter. Here is an example where none of the changes are related to this PRs goal:

Please remove unrelated changes if possible :-)

Thank you!

[Apollo3zehn]

One thing that also comes into my mind is, now with measured being the default, what happens if a string is larger than the maximum object header size? Will it corrupt the file? I don't have the spec available right now but IIRC there is an upper limit.

[Blackclaws]

Thanks — both points addressed, and the object header question turned out to be worth asking. Rebased onto current dev.

The unrelated changes are gone

The reformatting came from auto-formatting tooling and is removed. The feature commit is rebuilt rather than patched, so none of it is in the history any more. Across the PR, deleted lines are down from 557 to 43, and every one of the 43 is a line the feature actually replaces. DatatypeMessage.Writing.cs went from 137 hunks to the six places that pass the width through; AttributeTests@datatypes.cs from 199 changed lines to 19.

Your commit is applied unchanged apart from two conflict resolutions, both keeping the file's existing style over the reformatting: GetTypeInfoForVariableLengthString keeps named arguments (your NullTerminate fix is in), and the pre-existing <param> docs on H5WriteOptions stay one-liners, with your two new blocks added. Two of your hunks in AttributeTests@datatypes.cs disappeared because they only existed to undo the positional-argument churn.

You were right about the size limit, and there was a bug underneath it

An attribute message declares its own size in a two-byte field, so 65535 is the ceiling for the whole message. GetEncodeSize checked the data against 64 * 1024, which is neither the right budget nor the right ceiling:

• the name, datatype and dataspace are counted in the same two bytes and were not counted at all
• > 64 * 1024 let a data size of exactly 65536 through, which then became 0 as a ushort
• both (ushort) casts were unchecked

Sizes in the resulting gap wrapped, and ObjectHeader2.Encode allocates the object header from those sizes before writing the messages at their real length — so the header overran its own allocation. The attribute was not the only casualty: a file with a dataset and two attributes could not be opened, because the root group's header was corrupt. h5dump rejected it too.

This predates measured widths entirely — file.Attributes["a"] = Enumerable.Range(0, 16378).ToArray() is 65512 bytes of data and reproduces it on dev today. The window is narrow, a little over 20 bytes wide, and moves with the length of the attribute name, which is why it went unnoticed.

2c5fc344 computes the size once, exactly, in a width that cannot wrap, and tests the whole message against ushort.MaxValue. I kept it as its own commit in case you would rather take it separately.

Worth checking against the C library, which enforces the same rule: at default libver it also refuses, also counts the name, and its limit is stricter than ours — 65479 bytes against our 65500 — purely because version 1 object headers pad every component to 8 bytes where the version 2/3 messages PureHDF writes do not. It reads our boundary case back without complaint (h5py and h5dump both fine), so we are inside the format limit and they simply spend more of it. With libver="v108" or later the limit disappears, because the attribute moves to dense storage — a fractal heap instead of an object header message. PureHDF writes superblock 3 already, so there is no format obstacle to that, only the missing FractalHeap/BTree2 write path. Well out of scope here, but that is the shape of it if large attributes ever come up.

The one thing that was a genuine regression

Measuring stores the value inline, so it competes for those 65535 bytes. A string attribute over roughly 64 KB therefore failed — where under a zero DefaultStringLength it used to be variable-length and write at any size, the value living in the global heap for a fixed 16 bytes per element. A 100 KB string attribute wrote fine before this PR and threw after it.

10923b2e fixes that by falling back to variable-length when a measured width will not fit. It is the same argument as the null element: measuring is only taken where it costs nothing, and a width that does not fit costs the write. Sizes that used to work keep working.

Two boundaries on that, both deliberate and both worth your eye:

• A width from DefaultStringLength is not adjusted. It was asked for explicitly, so Inherit reports the limit rather than quietly writing a different datatype than the caller declared. The fallback belongs to the width the writer chose itself.
• A non-string attribute still throws, because there is nothing to fall back to. The only variable-length form for int[] is a variable-length sequence, which would turn an attribute of shape (n,) and type int32 into a scalar of type "sequence of int32" — a change to what a reader sees, not to where the bytes live. Refusing seemed better than that, and it is what the C library does.

AttributeMessageSizeTests covers all of it: the limit pinned from both sides one element apart, the name-length sensitivity, the fallback for a wide value and for a long array, the Inherit carve-out, the numeric case, and a file whose dataset and other attributes survive an oversized attribute.

[Apollo3zehn]

I found one scalar-null edge case that is not covered by the current tests.

The PR says measured string attributes fall back to variable-length when they contain null, because fixed-length strings cannot represent the difference between null and "". That works for string arrays with a null element, but not for scalar null string attributes.

Reproduction:

using PureHDF;

var file = new H5File();
file.Attributes["text"] = new H5Attribute<string?>((string?)null);

using var stream = new MemoryStream();
file.Write(stream);

Current result:

System.IndexOutOfRangeException: Index was outside the bounds of the array.

The reason appears to be that WriteUtils.ToMemory<T, TElement>(attribute.Data) returns an empty/default Memory<string> when attribute.Data is null. For a scalar null string attribute, that loses the distinction between “one scalar element whose value is null” and “no elements”. The measured string logic then never sees the null element, chooses a fixed width, and the scalar encoder later indexes source.Span[0].

Can you suggest how you would prefer to solve this? A narrow fix would be to preserve scalar null string attributes as one null element before measuring, so the existing null-element fallback chooses variable-length. But this may also point to a broader question around how scalar null reference-type attributes should be represented generally.

[Blackclaws]

Thanks — and the broader question you raise is the right one to ask, because the answer turned out to settle the narrow case too.

Two things before the fix itself. This is not caused by this PR: the same exception, at the same call site, reproduces on dev, and on this branch with AttributeStringLength.Inherit, so measured widths are not involved. And it is not really about strings. The common factor is a null dataspace meeting a scalar datatype, which a scalar encoder reads by indexing element zero:

new H5Attribute<string?>((string?)null)   // IndexOutOfRangeException
new H5Attribute<string>()                 // IndexOutOfRangeException
new H5Attribute<int>()                    // IndexOutOfRangeException
new H5Attribute<int[]>()                  // fine

new H5Attribute<int>() is the documented null-dataspace form with no string anywhere. int[] survives only because the array encoder walks a span rather than indexing. So there are two defects, and your narrow fix would have closed the first line and left the other two.

A null dataspace should carry no data. The size was derived from the dataspace dimensions, which a null dataspace has none of, and the empty product is one — so one element's worth was reserved and then asked for. The C library writes nothing here whatever the datatype: its attribute messages are 34 bytes for a null-dataspace int and 44 for a null-dataspace variable-length string, in both cases exactly name plus datatype plus dataspace. PureHDF was four bytes over on the int[] case that did not crash.

A scalar null is one element that is null. This is your broader question, and the C library answers it: H5Awrite accepts a null char* for a variable-length string and encodes it as a zero length and a zero global heap ID. That is the form PureHDF already writes for a null element inside an array, so a scalar null only has to reach it. Having done that, the 16 data bytes are byte-identical to the C library's, h5dump prints NULL rather than "", and PureHDF reads the C library's own file back as null — which h5py does not, since it flattens both to ''.

Both are fixed in #186, off dev, since neither belongs to this PR. Nothing further is needed here — this branch is unchanged and still only about string widths.

One incidental thing, since it cost me a test: new H5Attribute<string?>(default) binds to the null-dataspace constructor rather than the value one, so the value case needs the cast you used.