Skip to content

Check every public function against cross-platform reference values - #78

Merged
ctreffs merged 3 commits into
masterfrom
test/reference-values
Oct 17, 2026
Merged

ctreffs merged 3 commits into
masterfrom
test/reference-values

Conversation

@ctreffs

@ctreffs ctreffs commented Oct 17, 2026 •

Copy link
Copy Markdown
Member

Makes "identical up to rounding on every platform" a CI-checked guarantee instead of an assumption.

#77 compares FirebladeMath's portable implementations against Apple's simd, but only on Apple platforms, and only for functions that have both. The C math libraries the wrappers call (Apple libm, glibc, ucrt, musl/WASI) were never compared at all.

How it works

  • Registry (ReferenceFunctions.swift): all 241 public functions, each with an accuracy class and input domains. That's every top-level function plus the static functions and initializers of the matrix and quaternion types. Generic functions are registered in Float and Double, and eulerAnglesXYZ is included too.
  • Inputs: 24 deterministic inputs per function, drawn from function-appropriate domains: valid ranges, unit vectors and quaternions, rotation matrices, well-conditioned matrices. Scalar functions also get NaN, ±∞, ±0, ±1, tiny and huge values.
  • Reference data (ReferenceData.swift): make reference-values evaluates every function on macOS (default traits) and stores inputs and results as exact bit patterns. The data is one raw string, so it adds no compile-time cost.
  • Check: every platform that runs the tests evaluates the same inputs and compares:
Accuracy class Used for Requirement
exact sqrt, floor, copysign, mod, predicates, transpose, clamp, lerp, orthographic, … bit-identical
libm single C math library calls (sin, pow, atan2, …) ≤ 4 ulp
composite(.products) dot, cross, multiply, act rounding, scaled by the size of the products (cancellation-safe)
composite(.tight / .loose) normalization and conversions / inverses, trig, Euler angles, projections rounding-level relative tolerance
all classes NaN, ±∞, sign of zero must match exactly
  • Coverage: Scripts/check-reference-coverage.py (make check-reference-coverage, in the macOS CI job) fails if a public function isn't registered, or if a registration is stale.

Platforms: CI results

Platform C math library Reference values
macOS 6.1 / 6.3, with and without simd Apple libm ✅ all 248 match
iOS simulator 6.1 / 6.3 Apple libm ✅
Linux 6.1 / 6.2 glibc ✅ all 248 match
Windows 6.1 / 6.3 ucrt ✅ all 248 match
WASI 6.4.1 wasi-libc (musl) ✅ all 248 match: new, the job now runs swift test under WasmKit
Static Linux (Musl) musl build only: the static SDK ships without Swift Testing
Android Bionic build only: running needs a device or emulator

So "identical up to rounding" holds on every platform that can run tests: exact operations are bit-identical, C-library calls are within 4 ulp, and NaN, ±∞ and the sign of zero match exactly.

Test fixes found by running on WASI:

  • The input generators used Double(1 << 53). On wasm32 Int is 32 bits, so that was 0 and every generated value became ∞. They now use * 0x1p-53. FirebladeMath itself was not affected.
  • 33 existing FunctionTests spot checks compared C-library results for exact equality. WASI's tanhf(0.5) is 1 ulp away from Apple's, which is legitimate. Those checks now allow 4 ulp, matching the libm class; exact operations stay exact.

Verified locally

  • macOS with simd and portable (trait off): all 248 registrations match, so FirebladeMath's own code reproduces the simd-generated values.

  • Mutation check: I injected six bugs across the accuracy classes:

    • fract NaN handling;
    • the lerp formula;
    • a sign in Euler 123;
    • a 0.1% change in the perspectiveRH half-angle factor;
    • the pow2 base;
    • a sign in cross.

    All six are caught, each naming the affected functions.

  • Coverage check: removing one registration fails the check and names the function.

CI results for Linux, Windows and WASI follow below.

https://claude.ai/code/session_011WSUtoLptjea3s3iC5wTNC

FirebladeMath should produce identical results on every platform, up
to rounding. Until now nothing verified that across platforms: the
differential tests compare portable vs simd only on Apple platforms,
and the C math libraries (Apple libm, glibc, ucrt, musl/WASI) were
never compared at all.

All 241 public functions (top-level functions plus the static functions
and initializers of the matrix and quaternion types; generics in Float
and Double, and eulerAnglesXYZ) are registered with an accuracy class
and input domains. `make reference-values` evaluates them on 24
deterministic inputs each (plus NaN, ±inf, ±0, ±1, tiny and huge
values for scalar functions) on the reference platform (macOS, default
traits) and stores inputs and results as exact bit patterns in
ReferenceData.swift. Every platform that runs the tests compares its
results against them:

- exact: sqrt, floor, copysign, fmod, predicates, transpose, clamp,
  lerp, orthographic, ... must be bit-identical
- libm: single C math library calls within 4 ulp
- composite: rounding-level relative tolerance (products scale with the
  inputs, loose for inverses, trigonometry, Euler angles, projections)
- NaN, infinities and the sign of zero must always match exactly

Scripts/check-reference-coverage.py fails if a public function is not
registered or a registration is stale. The WASI job now runs the tests
instead of only building them.

Claude-Session: https://claude.ai/code/session_011WSUtoLptjea3s3iC5wTNC
Running the tests on WASI (new in this PR) showed two test problems:

- The pseudo-random input generators computed `Double(1 << 53)`. Int is
  32-bit on wasm32, so the shift produced 0 and every generated value
  became infinity (512 failed expectations). Multiply by 0x1p-53
  instead.
- FunctionTests compared C math library results (tanh, sin, pow, ...)
  for exact equality. WASI's libc returns tanhf(0.5) 1 ulp away from
  Apple's libm, which is legitimate. These 33 spot checks now allow
  4 ulp, matching the libm class of the reference-value tests; exact
  operations stay exact.

The reference-value test itself passed on WASI.

Claude-Session: https://claude.ai/code/session_011WSUtoLptjea3s3iC5wTNC
@ctreffs
ctreffs merged commit ba30ff5 into master Oct 17, 2026
16 checks passed
@ctreffs
ctreffs deleted the test/reference-values branch October 17, 2026 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant