Skip to content

Benchmark FirebladeMath's own implementations and gate allocations in CI - #75

Merged
ctreffs merged 1 commit into
masterfrom
ci/benchmarks
Oct 17, 2026
Merged

ctreffs merged 1 commit into
masterfrom
ci/benchmarks

Conversation

@ctreffs

@ctreffs ctreffs commented Oct 17, 2026 •

Copy link
Copy Markdown
Member

Adds runtime benchmarks for FirebladeMath's own implementations and a CI check that none of them allocate.

Thin wrappers around libm, simd or the standard library (sqrt, sin, dot, …) are deliberately not benchmarked, because they would only measure the platform library.

What

  • Benchmarks/ is a separate package using ordo-one/benchmark 1.38. It's the package from the swift.org benchmarks article, renamed since then from package-benchmark. Consumers of FirebladeMath never fetch it. It depends on FirebladeMath by path (..) and calls it across the module boundary, as a consumer does.

  • 15 benchmarks in two groups:

    Group Benchmarks
    Own code on every platform adjugate, Mat4x4f(rotation:axis:), look(from:at:up:), perspectiveRH, fromEulerAngles_123, quaternionToEulerAngles_123, the flat subscript, quaternion iteration
    Own code on the portable path only column construction, matrix multiply, inverse, matrix4x4(from:), quaternion(matrix:), quaternion multiply, act+normalize
    • init(rotation:axis:) and look(from:at:up:) are FirebladeMath's own algorithms, but they call the normalize/cross/dot/sin/cos wrappers internally.
    • With the simd trait (the default on Apple), the second group forwards to Apple's simd. Its numbers describe FirebladeMath's code only on the portable path: Linux and Windows, or locally with FRB_MATH_BENCHMARK_NO_SIMD=1.
  • Allocation check: Benchmarks/Thresholds/*.p90.json records the expected mallocCountTotal, which is 0 for every benchmark. The new Benchmarks workflow runs on Linux (swift:6.4.1), so it checks the portable path, and fails if any benchmark starts allocating. The simd path isn't checked in CI; run make benchmark-thresholds-check locally on a Mac for that. Timings are reported but not checked, because shared CI runners are too noisy.

  • Make targets: benchmark, benchmark-thresholds-check, benchmark-thresholds-update.

Results

M1 Max, Swift 6.4, p50:

Benchmark simd portable Allocations
Mat4x4f.adjugate 17 ns 17 ns 0
Mat4x4f.initRotation 8 ns 9 ns 0
Mat4x4f.lookAt 16 ns 13 ns 0
Mat4x4f.perspectiveRH 3 ns 3 ns 0
Quat4f.fromEulerAngles123 4 ns 4 ns 0
Quat4f.toEulerAngles123 10 ns 10 ns 0
Mat4x4f.flatSubscript 5 ns 5 ns 0
Quat4f.iteration 3 ns 3 ns 0
Mat4x4f.initColumns 2 ns 2 ns 0
Mat4x4f.multiply 9 ns 8 ns 0
Mat4x4f.inverse 10 ns 21 ns 0
Mat4x4f.fromQuaternion 6 ns 7 ns 0
Quat4f.fromMatrix 6 ns 7 ns 0
Quat4f.multiply 7 ns 13 ns 0
Quat4f.actNormalize 27 ns 27 ns 0

FirebladeMath's own portable inverse and quaternion multiply are about 2× slower than simd. They are the obvious candidates for optimizing the portable path.

Verified

  • The allocation check passes locally on macOS and in this PR's Linux CI run.
  • Negative test on this benchmark set: I temporarily made Mat4x4f.flatSubscript sum over .elements, which returns a heap Array. The check failed with "Deviations worse than threshold … Mat4x4f.flatSubscript" (p90 went from 0 to 1000) and a non-zero exit code.

Notes

  • Benchmark names contain no spaces or /. The benchmark tool writes threshold files with those characters replaced, but reads them back by the raw name, so such names never match. There's a comment in the code.
  • The repo folder must be named math. The benchmark package refers to FirebladeMath by its folder name, which is fine in CI and in a normal clone.
  • Adding a benchmark requires running make benchmark-thresholds-update to record its threshold.

https://claude.ai/code/session_011WSUtoLptjea3s3iC5wTNC

Add a separate Benchmarks/ package using ordo-one/benchmark (the package
from the swift.org benchmarks article), so the benchmark dependency
never enters FirebladeMath's dependency graph. Benchmarks call
FirebladeMath across the module boundary, like a consumer.

Only FirebladeMath's own implementations are benchmarked; thin wrappers
around libm, simd or the standard library are left out because they
would only measure the platform library:

- Own on every platform: adjugate, Mat4x4f(rotation:axis:),
  look(from:at:up:), perspectiveRH, Euler conversions, flat subscript,
  quaternion iteration.
- Own on the portable path only (forwards to simd on Apple): column
  construction, matrix multiply/inverse, matrix<->quaternion conversion,
  quaternion multiply, act+normalize.

Every benchmark measures mallocCountTotal. Benchmarks/Thresholds/
records 0 for all of them, and the new Benchmarks workflow (Linux,
portable path) fails if any benchmark starts allocating. Timings are not
gated. FRB_MATH_BENCHMARK_NO_SIMD=1 benchmarks the portable path on
Apple platforms. Make targets: benchmark, benchmark-thresholds-check,
benchmark-thresholds-update.

Claude-Session: https://claude.ai/code/session_011WSUtoLptjea3s3iC5wTNC
@ctreffs ctreffs changed the title Add runtime benchmarks with an allocation gate Benchmark FirebladeMath's own implementations and gate allocations in CI Oct 17, 2026
@ctreffs
ctreffs enabled auto-merge (squash) October 17, 2026 16:05
@ctreffs
ctreffs merged commit bbfd964 into master Oct 17, 2026
16 checks passed
@ctreffs
ctreffs deleted the ci/benchmarks branch October 17, 2026 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant