You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Qwen2/Qwen2.5 and Qwen3 are now defined by /pdsl/qwen2.pdsl and
/pdsl/qwen3.pdsl, from the residual stream to the vocabulary logits:
the layer loop, the final RMSNorm and the tied output projection.
Qwen3.model() builds from the file matching the checkpoint instead of
assembling the layers in Java. The embedding lookup remains in Java.
Language support for whole-model files:
- weights["..."] reads a weight from a checkpoint by name, failing with
the closest names when it is absent (StateDictionary.require)
- {name} in a string literal is replaced by the value bound to name
- zeros(shape) and rope_freqs(theta, head_size, seq_len) builtins
- buildModel binds a StateDictionary to the model's checkpoint
parameter and accepts compute requirements
The attention and transformer layers, and skytnt_block, take their KV
cache as explicit key_cache/value_cache parameters, replacing the
attention_cache state block: a called layer is built in the program
scope, so a state block would give every layer of a model one shared
cache. Java callers bind by name and are unchanged.
QwenModelFileTest checks both model files against the previous Java
layer assembly on synthetic checkpoints. The plan for this direction is
docs/plans/PDSL_FOR_RESEARCH_AND_EDUCATION.md.
A model now repeats its layers with stack, which builds its body once
per member of a weight group in numeric order, with no index the body
can compute with:
stack weights.model.layers as block { ... block.mlp.up_proj.weight ... }
A field access on a checkpoint is a path into it: weights.model.norm.weight
is that tensor and weights.model.layers is the group under it. A missing
path fails with the closest names. Numbered members are reachable only
through stack. Fields may be spelled like keywords (weights.model).
qwen2.pdsl and qwen3.pdsl use stack and paths in place of the for loop,
the {i} string substitution and weights["..."] string keys, which are
removed from the language. StateDictionary gains group() and members().
PdslStackTest covers stack and weight paths. CLAUDE.md notes that the
ar-manager tools may surface under either of two prefixes.
new StateDictionary(directory) now reads a published checkpoint's
.safetensors files without a Python conversion step. When a directory
holds any safetensors file, only those are read and its configuration
and tokenizer files are ignored.
SafetensorsReference locates each tensor from the file's JSON header
(BF16, F16, F32 or F64) and, like protobuf weights, leaves the values in
the file until something reads them, through the same FileMapping.
Decoding a stored value moves onto CollectionDataReference.valueAt, which
SafetensorsReference overrides for its element types.
Safetensors is read only; weights are still saved as protobuf. A file is
mapped whole, so one file can be at most 2 GB.
SafetensorsReferenceTest writes a small checkpoint of every supported
element type and reads it back exactly.
SafetensorsReference called Float.float16ToFloat(short), and
SafetensorsReferenceTest called Float.floatToFloat16(float); both were
added in JDK 20, but the project targets Java 17, so ar-ml failed to
compile and no tests could run.
Add Java-17-compatible IEEE-754 half-precision conversions
float16ToFloat(short) and floatToFloat16(float) to org.almostrealism.io.Bits
(the existing home for primitive bit manipulation, mirroring the JDK's own
static Float methods) and call them from the F16 decoder and the test.
float16ToFloat is lossless and carries subnormals, infinities and NaNs;
floatToFloat16 rounds to nearest with ties to even and saturates to a
signed infinity on overflow. Extend BitsTest with roundtrip, smallest
subnormal, special-value and overflow coverage.
…tings
SafetensorsReference.locate now rejects a tensor whose byte range is
reversed, negative or extends past the file's tensor data, and a tensor
with a negative axis, when the header is read rather than when the
tensor is first used. A tensor with a zero-length axis is still left out.
New tests cover these rejections, StateDictionary member ordering,
groups and require() suggestions, models without a checkpoint
parameter, stacks over empty groups, zeros(), the rejection of untied
output weights by Qwen3, and the PDSL settings of Qwen3Config.
The no-checkpoint guard compares map sizes, so an extraArgs entry that happens to use the declared checkpoint parameter name makes args.size() == extraArgs.size() even though the checkpoint was successfully bound (and overwritten). This produces the misleading "declares no checkpoint parameter" error for a valid model declaration; test for whether a checkpoint parameter was found instead of using map-size changes.
Implementation status contradicts documented completed features
docs/plans/PDSL_FOR_RESEARCH_AND_EDUCATION.md:3
The status says that nothing in this plan has been implemented, but this same document now marks stack/weight binding and the pure-Java safetensors reader as implemented. That contradiction makes the plan's implementation status unreliable; update the status to reflect the partial implementation.
Address the three findings from the latest review of the PDSL checkpoint work:
- SafetensorsReference.locateTensor validated data_offsets only for
non-empty tensors: a zero-length axis returned null before the range
check ran, so a malformed empty entry (a non-empty or out-of-range
data_offsets) was silently omitted rather than rejected, breaking the
contract locate documents. Validate the range for every tensor, then
require an empty tensor's range to itself be empty before omitting it.
- PdslInterpreter.buildModel detected a missing checkpoint parameter by
comparing argument-map sizes; when extraArgs already held a key equal
to the checkpoint parameter's name, binding overwrote it instead of
growing the map, so a valid model falsely reported "declares no
checkpoint parameter". Track whether a checkpoint parameter was bound.
- Correct the plan document's status line, which claimed nothing had been
implemented while the body marks stack, hierarchical weights, explicit
KV caches and the safetensors reader as done.
Add tests covering an empty tensor with a non-empty range, an empty tensor
with out-of-range offsets, and checkpoint binding over a colliding extra
argument.
This new safetensors branch calls logLoaded, whose message hard-codes "protobuf files". Loading a Hugging Face directory therefore reports safetensors tensors as coming from protobuf files, making the startup diagnostic misleading. Emit a safetensors-specific message here or parameterize the helper by file format.
Validate safetensors headers and normalize parse errors
The documented malformed-header contract is not upheld here: missing dtype, shape, or data_offsets fields can produce NullPointerException/IndexOutOfBoundsException, and invalid JSON/root types escape as Gson parse/state exceptions rather than IllegalArgumentException. Since StateDictionary now opens external checkpoint files through this method, validate the required fields and wrap parse/type failures in an IllegalArgumentException that names the file or tensor.
Address the three findings from the latest review of the PDSL-for-research
branch.
- SafetensorsReference.locate/locateTensor now uphold their documented
"malformed header -> IllegalArgumentException" contract: a header that is
not a JSON object, a tensor not described by a JSON object, and missing
dtype/shape/data_offsets fields (plus a data_offsets that is not a
[begin, end] pair) are each rejected with a message naming the file or
tensor, instead of escaping as a Gson parse/state exception, an NPE, or an
IndexOutOfBoundsException.
- StateDictionary.logLoaded is parameterized by on-disk format so loading a
safetensors checkpoint no longer reports its tensors as coming from
protobuf files.
- The Phase 2 plan status no longer says a checkpoint must be converted to
protobuf; the conversion is now optional, since StateDictionary reads
safetensors directly.
Adds SafetensorsReferenceTest cases covering each new rejection path.
These JSON conversions still let structurally malformed tensor entries escape as Gson IllegalStateException (for example, shape or data_offsets being a scalar instead of an array), because only the non-object entry is wrapped above. locate documents IllegalArgumentException for malformed headers and the new tests establish that callers receive a file/tensor-specific message; validate the JSON types/conversions here or wrap their failures and rethrow the same diagnostic exception.
…ception
A safetensors header field that was present but of the wrong JSON type (a dtype
that is not a string, a shape or data_offsets that is not an array of integers)
escaped SafetensorsReference.locate as one of gson's own runtime exceptions,
breaking the IllegalArgumentException contract its javadoc promises. require()
now converts the field itself and reports a malformed field by name, and shape
and data_offsets are read as exact longs, so a fractional, non-numeric or
nested value is rejected and an axis beyond the int range is reported rather
than silently truncated by getAsInt.
A truncated file with fewer than 8 bytes reaches readFully and escapes as EOFException, even though locate documents malformed headers as IllegalArgumentException and the later truncated-header path normalizes that failure. Check the file length before readFully (or normalize this exception) so malformed safetensors files have one consistent failure contract.
This issue also appears on line 233 of the same file.
…ntException
SafetensorsReference.locate now rejects a file too short to hold the
8-byte header length as not a safetensors file, instead of letting an
EOFException escape. It also rejects a header longer than
MAX_HEADER_LENGTH before allocating the array that holds it, so the
length cannot overflow when cast to int. The file length is read once
and reused.
New tests cover a file of 0, 1 and 7 bytes, a file holding only the
header length, and a sparse file whose header length goes past the
maximum.
The safetensors header schema requires shape and data_offsets to be arrays
of JSON integers. SafetensorsReference.longs() read each element with
getAsBigDecimal(), which silently parses a quoted numeric string such as
"4", so a header quoting its shape or byte ranges passed as well-formed.
longs() now requires each element to be a JSON number before converting it,
upholding the method's documented "array of integers" contract alongside the
existing checks for non-integer and out-of-range values. Added
SafetensorsReferenceTest cases covering a quoted axis and quoted offsets.
StateDictionary.group() returned a dictionary that shared the root's
PackedCollection instances, while destroy() unconditionally destroyed
every value in its map. Destroying a derived (or nested) group therefore
invalidated the root dictionary and every sibling group that shared those
weights, so later inference could read released storage. Groups are
created freely by the PDSL interpreter (stack members and checkpoint path
access), making this a live hazard.
Add an owning flag to StateDictionary: file/asset/map constructors own
their tensors, and group() now produces a non-owning view. destroy() only
releases the tensors when the dictionary owns them, so destroying a group
clears its own view without touching the shared weights, while destroying
the owning root still releases them. Document the behaviour on group()
and destroy().
Add PdslStackTest.destroyingGroupLeavesRootWeightsIntact covering both
directions: a group's destroy() leaves the shared tensor and the root
intact, and the owning root's destroy() releases the shared tensor.
The test header contains BF16, F16 and F32 tensors, but no F64 tensor despite the test's claim that it has one of each element type. Consequently the new Encoding.F64/decode path is untested. Add an F64 entry with its offsets and assert its decoded value so all supported encodings are covered.
SafetensorsReference.locate now collects every tensor's byte range,
including those of tensors with a zero-length axis, and checks that,
in order of position, they begin at zero, each begins where the
previous ended, and the last ends at the end of the file. Overlapping
ranges (which would expose the same bytes as two weights), gaps and
trailing bytes are rejected with IllegalArgumentException, as the
safetensors format itself requires.
Tests cover overlapping and identical ranges, gaps, trailing bytes,
ranges listed out of order, an empty header, and reading F64 values.
A sharded checkpoint assigns each tensor to exactly one shard, and the
order the shards are read in is not part of that contract. StateDictionary
now records which shard each tensor came from and rejects a name an
earlier shard already defined, naming both files, before adding any tensor
of the later file. Before, the filesystem's ordering decided which of the
two definitions survived.
SafetensorsReferenceTest covers reading a two-shard directory and
rejecting a tensor that both shards define, whether or not the two
definitions agree.
A StateDictionary whose load fails part way through is never returned to a
caller, so nothing could destroy it and the shards it had already mapped
stayed mapped. init() now destroys the partially loaded dictionary before
rethrowing. loadWeights() also resolves each asset's file once instead of
once per pass, since resolving an asset verifies its checksum and may
download it.
Tests cover both a failed load (a redefined tensor and a malformed shard)
and destroying a dictionary read from shards, checking the mapped file count
returns to its baseline.
SafetensorsReference reads the declared header length from the file before
parsing any of the header. An untrusted file could name a length approaching
MAX_HEADER_LENGTH and, if the file were long enough (a sparse file costs almost
no disk), force a multi-gigabyte allocation that exhausts the heap before a
single byte is validated.
Add MAX_SAFE_HEADER_LENGTH (100 MB, matching the reference implementation's
limit) and reject any declared header larger than it before the buffer is
allocated. The existing MAX_HEADER_LENGTH remains the format's technical
ceiling. A genuine checkpoint header is a small JSON object and comes nowhere
near the cap.
Covered by SafetensorsReferenceTest.rejectsHeaderBeyondSafetyCap, which
declares an over-cap header in a sparse file and asserts the rejection names
both the header size and the allocation limit.
AudioSceneTestBase.requireCuratedLibrary failed any GPU host missing the
curated sample library. The Metal test-media-mac job runs the pipeline
profile without declaring AR_RINGS_LIBRARY, so the two curated-library
tests that do not exclude the pipeline profile (GenerateAudioFileTest,
AudioSceneOptimizerStemTest) failed whenever a branch triggered that lane.
The helper now Assume-skips on a pipeline run that declares no library
mount, matching the sibling curated tests that exclude that profile. A
runner that declares the mount (test-media-cl) still fails if it is lost.
The reason will be displayed to describe this comment to others. Learn more.
Copilot review overview
🔵 Needs a closer look
The broad PDSL, model-inference, checkpoint-loading, and precision changes require final human review.
Review effort: Lite Findings: None
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.