[WIP] Preset patching, per-trial records, and --previous - #4118
Draft
peterschmidt85 wants to merge 8 commits into
Draft
[WIP] Preset patching, per-trial records, and --previous#4118peterschmidt85 wants to merge 8 commits into
peterschmidt85 wants to merge 8 commits into
Conversation
A trial was one appended line in a shared `trials.jsonl`, mirrored into the session directory by byte offset. An agent rewriting the file moved every offset, and the mirror then committed a torn fragment forever: one verified preset still carries an unparseable trial record. Service attempts had the same shape in `verifications.jsonl`. Every record is now written exactly once and reading is listing a directory: - `trials/<n>/` holds the trial's compiled `task.dstack.yml` and its `trial.json`; the presence of `trial.json` is what marks the trial completed, so in-flight and torn states are visible instead of corrupting. - `service/<k>/` holds each verification attempt's submitted YAML and a `verification.json` written when the attempt ends; an attempt directory without a result is one still in progress. - The byte-offset record mirrors are replaced by a stateless directory mirror that re-lists the source and copies changed files whole, scrubbed, and atomically. A torn read can never be committed; the next pass converges. Trial and attempt directories sort numerically, pinned past 9. The trial contract in the agent prompt is rewritten around the layout, with every per-trial file referenced by its full path and the write order stated where the files are defined. The listing shows the trial being worked on rather than the completed count while trialing, so `trialing (2/3)` cannot read as two finished; `verifying (3/3)` keeps the completed count. Validated end to end: a session on one RTX PRO 4500 produced three contract-exact trial records, a recorded verification attempt, and a saved preset that `dstack preset apply` deploys. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The directory mirror decoded each file to text and re-encoded it, which rewrote newlines (CRLF became LF on Unix, LF would become CRLF on Windows) and replaced non-UTF-8 bytes. Harmless for the CLI's own JSON and YAML records, but the mirror now also carries patch files, where the recorded copy must be exactly the bytes that ran. Copy bytes verbatim and redact at the byte level. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The preset agent may now patch the serving framework source code, generate custom kernels, and patch drivers during trials. Patches are recorded as unified diffs under `trials/<n>/patches`, referenced from the trial's `task.dstack.yml` via `files`, and applied with `patch` from its commands, so the recorded configuration reproduces the trial exactly. The verified service records its patches the same way under `service/<k>/patches`, and the saved preset re-roots `files` onto the mirrored session copies, so `dstack preset apply` reproduces the patched configuration after the agent workspace is deleted. When no trial meets the constraints, the best failed trial that has a benchmark is still verified and saved instead of failing the whole creation: the hardware's best is worth keeping even when it falls short of what was asked. To keep that honest, the final report now records the verified trial, presets store the requested `min_context_length` and `max_ttft` alongside the verified values, and `dstack preset apply` warns, instead of refusing, when the preset's verified context is below the requested one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A preset saved from a failed trial shows `*` next to its benchmark, the same mark a running session uses when it has only failed trials to show, computed from the requested constraints the preset now stores. In the trial spark, bars scale from zero so their heights compare as the numbers do, failed trials are red rather than gold, and gold marks the best result only while no trial meets the constraints. While trialing, the `(N/M)` progress counts the trial being worked on rather than the completed ones. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The system prompt rendered conditional blocks with a non-nesting `<!--?NAME:CONTENT-->` directive. Replace it with `<!--?if NAME-->` / `<!--?else-->` / `<!--?end-->`: blocks nest, markers may be inline or alone on a possibly indented line, and a branch body is dedented by the one indentation shared by all its lines. Both branches are always parsed, so an unknown variable cannot hide behind a flag combination, and unbalanced or malformed directives fail loudly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
peterschmidt85
changed the title
peterschmidt85
force-pushed
the
preset-trial-folders
branch
from
August 11, 2026 19:30
24ebce2 to
4e44be5
Compare
`dstack preset create --previous <ID>` (repeatable, or a `previous` list in the configuration) gives the agent the records of earlier creation sessions: their trials, configurations, patches, and reports are copied into the workspace under `previous/`, and the prompt tells the agent to analyze them and improve on them. With `baseline: true`, the first trial reproduces the best comparable previous result before optimizing further. The IDs are pinned in the session manifest, so a resumed session keeps the same context. A still-running previous session is rejected; a chained session whose parents were not included warns. `constraints.json` is now written as a session record for every creation, not only under `--debug`, so a preset's constraints survive after its workspace is deleted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
peterschmidt85
force-pushed
the
preset-trial-folders
branch
from
August 11, 2026 19:37
4e44be5 to
6d56b96
Compare
CONSTRAINTS and BENCHMARK now wrap together to keep their full content, STATUS and SUBMITTED wrap at their spaces, and BASE and GPU truncate (capped in the compact view) so a long model name cannot starve the data columns. Previously only BENCHMARK folded while its neighbours clipped, so a narrow terminal rendered one tall column beside single-line ones. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
peterschmidt85
force-pushed
the
preset-trial-folders
branch
from
August 11, 2026 21:02
84ce68e to
bfdf732
Compare
Note in the concepts intro that the agent optimizes across the serving stack and may patch the framework's source, generate kernels, and patch drivers. Replace the roadmap admonition with a limitations one that also records the random-dataset and runtime-only-patch limits. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
peterschmidt85
force-pushed
the
preset-trial-folders
branch
from
August 11, 2026 21:03
bfdf732 to
8450775
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three improvements to
dstack preset create:dstack preset applyreproduces them.*indstack preset. Applying it warns when its context length is below what was asked for.dstack preset create --previous <ID>(repeatable, or apreviouslist in the configuration) gives the agent the results of earlier sessions to analyze and improve on, instead of starting from scratch. Works with presets created by this version or later.To be done:
🤖 Generated with Claude Code