Add file-level user metadata#8740
Draft
mprammer wants to merge 2 commits into
Draft
Conversation
Adds optional file-level metadata to the Vortex file format: string-keyed, opaque byte segments referenced from the postscript. Keys and locators live in the postscript (read at open); values load only on `include_metadata`, resolved per-locator through the segment source (from the initial footer read when covered, else a targeted read) and copied out. A default open materializes nothing. `Footer` carries only locators, so a cached footer is identical regardless of the opener's flag and one file's metadata can't surface for another via the multi-file cache. `DType`, its FlatBuffers/protobuf serialization, and the scan path are untouched; the wire change is a single additive `Postscript` field (file version stays 1). `Alignment::from_exponent` on untrusted postscript input now returns an error instead of panicking (the TUI inspector included). Revives Nicholas Gates' file-metadata-segments work (#7954), rebased onto develop and reshaped so metadata never widens the footer read or pins its backing buffer. cargo build + fmt + clippy clean; vortex-file 100/100, vortex-buffer 822/822. Co-Authored-By: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: mprammer <martin@spiraldb.com>
Merging this PR will not alter performance
Comparing 
Footnotes
|
robert3005
reviewed
Jul 14, 2026
| footer: PostscriptSegment; | ||
| /// User-defined metadata segments keyed by string. Keys must be unique, non-empty, and at most | ||
| /// 32 UTF-8 bytes; readers reject postscripts that violate these limits. | ||
| metadata: [PostscriptMetadata]; |
Contributor
There was a problem hiding this comment.
I think we cannot put keys into the postscript. Imho this should just be another segment that readers can choose to read. We should likely make it stacked above schema
Move user metadata out of the postscript. Instead of per-key [PostscriptMetadata] locators embedded in the postscript, the postscript now carries one optional locator to a FileMetadata segment (keyed opaque values) that may live anywhere in the file. Keys no longer pressure the postscript, so the count/length limits are dropped. A default open reads no metadata; include_metadata resolves the single segment in one read. Addresses review feedback on the original approach. Co-Authored-By: Nicholas Gates <nick@nickgates.com> Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: mprammer <martin@spiraldb.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds optional file-level metadata to the Vortex file format: a set of string-keyed, opaque byte segments referenced from the postscript, for consumers that need to attach identity or annotation to a file (Iceberg field IDs, an Arrow-metadata round-trip, provenance). Keys and their segment locators live in the postscript and are read at open; the opaque values load only when a reader opts in with
include_metadata, resolved one locator at a time through the file's existing segment source — served from the initial footer read when they fall inside it, otherwise a targeted read. A default open never materializes them and makes no metadata-driven read. TheDType, its FlatBuffers and protobuf serialization, and the scan path are unchanged; the wire change is a single additivePostscriptfield, so old readers skip it and the file version stays1.This revives #7954 (the original file-metadata-segments work) rebased onto develop, with the read path reshaped so metadata is never folded into the footer's contiguous tail read and never keeps that buffer alive: values are resolved per-locator and copied out.
Footercarries only the locators, so a cached footer is identical whether or not the opener asked for metadata, and one file's metadata can't surface for another through the multi-file cache. Parsing a segment alignment from an untrusted postscript now returns an error rather than panicking on an out-of-range exponent (the TUI inspector included). The public read API is additive —VortexFilegains metadata accessors andVortexOpenOptionsaninclude_metadataopt-in — andFooter::newis unchanged; the generatedPostscriptFlatBuffer gains a field, as any additive schema change does.🤖 Generated with Claude Code