Skip to content

Move the benchmarks to scaling 0.9, and compare with the previous release - #66

Open
droundy wants to merge 3 commits into
masterfrom
scaling-0.9
Open

droundy wants to merge 3 commits into
masterfrom
scaling-0.9

Conversation

@droundy

@droundy droundy commented Oct 11, 2026

Copy link
Copy Markdown
Owner

Summary

scaling 0.9 has no bench / bench_gen_env, so benches/bench.rs (the only user of scaling) is rewritten, as four binaries:

  • versions: this release against the previous published one (internment_old, 0.8.6) at 1% accuracy, with old as the baseline of every group.
  • controls: each release against itself, so there is a noise floor to judge versions by.
  • libraries: Intern, NewIntern, ArcIntern, Intern32, arc-interner and plain Strings, to within 10%.
  • contended: the old 100000-value workloads, one group per workload and contention level, one candidate per interner.

Run with cargo bench --features benchmarks --bench versions (README has a new section).

Design

  • scaling is an optional dependency, not a dev-dependency. It needs Rust 1.81 and cargo test builds dev-dependencies, so a dev-dependency fails cargo +1.71 test --no-run. internment_old is optional too (a dev-dependency cannot be).
  • Two features. bench is unchanged and stays a leaf (it only exposes benchmarking_only_clear_interns); the new benchmarks adds scaling, internment_old and arc-interner. This matters for later: once this version is "the previous release", the next one enables its bench, and that must not pull in scaling and the release before it, and so on down. The split has to ship with the first release that has these benchmarks.
  • Each release then bumps the internment_old pin; plans/scaling-0.9.md describes that cycle and the reasoning.
  • Fixes the old bench, which did not compile on master (bench did not enable arc-interner).

Behaviour changes in what is measured

  • compare/hash is now lookup: the old two sets had nothing in common, so its removals never found anything. lookup does the same hashing and comparing read-only, which lets the sets be built once, untimed.
  • sort sorts references (so a String baseline is not charged for copying) and starts from a fixed shuffled order; the old one got a random order from HashSet iteration.
  • Intern32 is a new candidate wherever the others run.

Also

  • src/intern32.rs: one #[cfg(feature = "deepsize")] on an impl whose bound named deepsize unconditionally, so --features intern32 compiles.
  • CI: a job that runs cargo check --benches --features benchmarks on stable.

Not done / known

  • The CI msrv job's cargo test very likely fails today from a fresh resolve (criterion 0.4 pulls in half 2.7.1, needing 1.81, and textwrap 0.16.4, needing 1.88), independent of this change; I checked with scratch copies but could not see recent CI runs.
  • cargo check --no-default-features --features intern and --features intern32 already fail on master.
  • controls separate the benchmark code from the library but do not build a second copy of the previous release as its own package.
  • Numbers are not committed (machine-dependent). A short run of versions already shows differences from 0.8.6, e.g. Intern<String> sort about +12% and Intern<i64>::new random about -18%, which were not measured at full accuracy.

Test plan

  • cargo check --benches --features benchmarks (stable)
  • cargo test --all-features -- --test-threads=1 (stable)
  • cargo +1.71 check, with and without --features arc, from a fresh lockfile
  • each of versions, controls, libraries and contended (the latter filtered) run briefly and produce results
  • a full contended run on a machine with reserved or unpinned CPUs

🤖 Generated with Claude Code

droundy and others added 2 commits October 10, 2026 19:32
Without it, --features intern32 failed to compile with an unresolved
deepsize crate, which the benchmarks need.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
scaling 0.9 replaces bench/bench_gen_env with declared benchmarks, groups
and baselines, so benches/bench.rs is rewritten as four binaries:

- versions: this release against the previous one (internment_old, 0.8.6),
- controls: each release against itself, for the noise floor,
- libraries: the interners against each other, String and arc-interner,
- contended: the 100000-value workloads, from one thread and from fifty.

scaling needs Rust 1.81, so it and internment_old are optional
dependencies behind a new benchmarks feature rather than dev-dependencies,
which the 1.70 test job would build.  bench stays a leaf feature (it only
exposes benchmarking_only_clear_interns), so that when this version is the
previous release the next one's benchmarks do not pull in a chain of every
release before it.  This also fixes the old bench, which did not compile
because bench did not enable arc-interner.

Adds a CI job that builds the benchmarks, a README section on running them,
and plans/scaling-0.9.md, which says why and how to carry on each release.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 11, 2026

Copy link
Copy Markdown

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

scaling 0.9.1 accepts `ty` fragments of macro_rules! in a benchmark's
signature, so vs! and twins! take ordinary types instead of token trees.

Also fixes the warnings `cargo bench --features benchmarks` printed:
the unused macros and import in the benchmark binaries, the elided
lifetimes in arena.rs, the unused TestStruct and TestStructCount in the
tests, and an import of println that was only needed with the tinyset
feature.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant