Measuring the submission step: agents often ship a configuration they never benchmarked

[ElnatanAnbelu]

Thanks for publishing the trajectories. Being able to re-analyse 269 runs without an H100 is
what made this possible at all.

The paper localises the deficit as "the ability to propose diverse configurations, evaluate them
systematically, and submit the best identified solution", supported by qualitative reading. I
tried to turn the third clause into something measurable, since it needs no baseline: of the
configurations an agent measured with its own evaluator, did it submit the winner?

Across the 33 runs where an agent measured two or more configurations at comparable scope:

submitted the best one they measured	8 (24%)
submitted one within 2% of it	2 (6%)
submitted a measured configuration that was clearly worse	8 (24%)
submitted a configuration they never measured	15 (45%)

Restricting to readings whose full scope is provable (27 runs) gives the same shape: 22% best,
15% worse, 56% unmeasured.

The split that surprised me is the last row. Those runs are not choosing badly among their
results, they are editing start_server.sh after their final measurement and stopping.
run_0138 is the clearest: last measurement at trace step 214, adds kv-cache-dtype=fp8 at step
219, run ends at 221. run_0006 does the same after fourteen measurements.

Three questions where you would know better than I can infer:

1. Does that match what you saw reading the trajectories, or is it an artefact of how I am
   reconstructing configurations? I rebuild start_server.sh across each run and take a
   signature at each launch, which is the only harness-independent handle I found.
2. Are the per-scenario PyTorch baselines published anywhere? metrics.json carries raw
   latency and throughput, so I cannot express outcomes as speedups, and without that a run
   that never launched a server can look good because the default one answered.
3. Is request_count recoverable for readings where the trace truncated before it? Scope is
   the binding constraint: 440 of 1,423 evaluator outputs in the corpus are four-request smoke
   tests, and excluding them correctly is what took the judgeable set from 79 runs to 33.

Code and the validation notes are here, including the four defects that hand-checking twenty
runs against the raw traces turned up: https://github.com/ElnatanAnbelu/agent-search-metric

Happy to send a PR against the paper's analysis, publish this separately, or drop it if it is
measuring an artefact.