Repository navigation
update benchmark suite - #658
Conversation
At n = 1 and 10, each potential benchmark measures only the fixed cost of a call (unit handling and validation), and n = 1,000 is still mostly fixed cost for cheap potentials. Replace n = 10 with n = 10,000 so the per-point cost of each evaluation path shows. The classes with expensive potentials keep their explicit size caps. Generate the positions in a fixture, so that the timed region is the evaluation alone: at n = 10,000, generating them costs about as much as evaluating a cheap potential. With a fixed seed per test, every test also gets the same positions, regardless of which tests ran before it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hessians had no benchmark. One cheap potential is enough to track the cost of the evaluation machinery around the kernels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both integration benchmarks took an n_orbits parameter but always integrated a single orbit, so each case ran four times under different names. Integrate n_orbits copies of the initial condition, and skip the cases with more than 10^6 orbit-steps, which are too slow under CodSpeed's simulation. Build the initial conditions in fixtures, so that only the integration is timed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The potential benchmarks pass plain arrays already in the potential's units, so they skip the conversions applied to other inputs. Benchmark energy with a Quantity, with an integrated Orbit, and through Orbit.energy(). The orbit has 10^6 points: at 10^4, the copies made converting it cost a few microseconds, hidden by the ~0.3 ms of fixed overhead per call, while at 10^6 they take most of the time: Orbit.energy() takes 69 ms, against 15 ms for the same positions as an array. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The O2GF benchmarks with 10 orbits took 49 and 33 minutes of the CodSpeed run, about 300 times their real time (11 s and 5.5 s), because most of O2GF's time is the Python-level fit of the toy potential. Each orbit is fit separately, so the cost is linear in the number of orbits, and timing 10 orbits measures the same code as timing one. Skip the multi-orbit cases under CodSpeed. They still run in the walltime scaling benchmarks, which plot the scaling with n_orbits in the docs and take about a minute. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merging this PR will degrade performance by 82.28%2 warnings
Performance Changes
Tip Curious why performance improved? Comment Comparing 
Footnotes
|
|
All the benchmark "regressions" are on the orbit integration, where we fixed the bug with Benchmark CI time is down to 15 minutes from 1+ hours, too. |
|
Regressions: acknowledged! |
Describe your changes
This is a collection of small updates to the benchmark suite in preparation for #477. The idea is to get most of the benchmarking in place that we hope will show performance improvements in the transpose. Some bugfixes, some expanded coverage. Codspeed will hopefully be faster by doing fewer O2GF orbits; it was dominating the time.
The different-array-types benchmarking is anticipating improvements in reducing the number of copies when the data is a Quantity or Orbit versus a plain array. But this could be something we test via a one-time benchmark if we don't want it in the suite.
Checklist
CHANGES.rst)