Repository navigation
Benchmark FirebladeMath's own implementations and gate allocations in CI - #75
Merged
Merged
Conversation
Add a separate Benchmarks/ package using ordo-one/benchmark (the package from the swift.org benchmarks article), so the benchmark dependency never enters FirebladeMath's dependency graph. Benchmarks call FirebladeMath across the module boundary, like a consumer. Only FirebladeMath's own implementations are benchmarked; thin wrappers around libm, simd or the standard library are left out because they would only measure the platform library: - Own on every platform: adjugate, Mat4x4f(rotation:axis:), look(from:at:up:), perspectiveRH, Euler conversions, flat subscript, quaternion iteration. - Own on the portable path only (forwards to simd on Apple): column construction, matrix multiply/inverse, matrix<->quaternion conversion, quaternion multiply, act+normalize. Every benchmark measures mallocCountTotal. Benchmarks/Thresholds/ records 0 for all of them, and the new Benchmarks workflow (Linux, portable path) fails if any benchmark starts allocating. Timings are not gated. FRB_MATH_BENCHMARK_NO_SIMD=1 benchmarks the portable path on Apple platforms. Make targets: benchmark, benchmark-thresholds-check, benchmark-thresholds-update. Claude-Session: https://claude.ai/code/session_011WSUtoLptjea3s3iC5wTNC
ctreffs
force-pushed
the
ci/benchmarks
branch
from
October 17, 2026 16:03
a116916 to
e0905ea
Compare
ctreffs
enabled auto-merge (squash)
October 17, 2026 16:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds runtime benchmarks for FirebladeMath's own implementations and a CI check that none of them allocate.
Thin wrappers around libm, simd or the standard library (
sqrt,sin,dot, …) are deliberately not benchmarked, because they would only measure the platform library.What
Benchmarks/is a separate package usingordo-one/benchmark1.38. It's the package from the swift.org benchmarks article, renamed since then frompackage-benchmark. Consumers of FirebladeMath never fetch it. It depends on FirebladeMath by path (..) and calls it across the module boundary, as a consumer does.15 benchmarks in two groups:
adjugate,Mat4x4f(rotation:axis:),look(from:at:up:),perspectiveRH,fromEulerAngles_123,quaternionToEulerAngles_123, the flat subscript, quaternion iterationmultiply,inverse,matrix4x4(from:),quaternion(matrix:), quaternionmultiply,act+normalizeinit(rotation:axis:)andlook(from:at:up:)are FirebladeMath's own algorithms, but they call thenormalize/cross/dot/sin/coswrappers internally.simdtrait (the default on Apple), the second group forwards to Apple's simd. Its numbers describe FirebladeMath's code only on the portable path: Linux and Windows, or locally withFRB_MATH_BENCHMARK_NO_SIMD=1.Allocation check:
Benchmarks/Thresholds/*.p90.jsonrecords the expectedmallocCountTotal, which is 0 for every benchmark. The new Benchmarks workflow runs on Linux (swift:6.4.1), so it checks the portable path, and fails if any benchmark starts allocating. The simd path isn't checked in CI; runmake benchmark-thresholds-checklocally on a Mac for that. Timings are reported but not checked, because shared CI runners are too noisy.Make targets:
benchmark,benchmark-thresholds-check,benchmark-thresholds-update.Results
M1 Max, Swift 6.4, p50:
Mat4x4f.adjugateMat4x4f.initRotationMat4x4f.lookAtMat4x4f.perspectiveRHQuat4f.fromEulerAngles123Quat4f.toEulerAngles123Mat4x4f.flatSubscriptQuat4f.iterationMat4x4f.initColumnsMat4x4f.multiplyMat4x4f.inverseMat4x4f.fromQuaternionQuat4f.fromMatrixQuat4f.multiplyQuat4f.actNormalizeFirebladeMath's own portable
inverseand quaternionmultiplyare about 2× slower than simd. They are the obvious candidates for optimizing the portable path.Verified
Mat4x4f.flatSubscriptsum over.elements, which returns a heapArray. The check failed with "Deviations worse than threshold … Mat4x4f.flatSubscript" (p90 went from 0 to 1000) and a non-zero exit code.Notes
/. The benchmark tool writes threshold files with those characters replaced, but reads them back by the raw name, so such names never match. There's a comment in the code.math. The benchmark package refers to FirebladeMath by its folder name, which is fine in CI and in a normal clone.make benchmark-thresholds-updateto record its threshold.https://claude.ai/code/session_011WSUtoLptjea3s3iC5wTNC