Repository navigation
Faster microfloat conversions: generated bit-twiddles and branch-free narrowing - #35
Merged
Merged
Conversation
@cvt_table now fits the cvt_generic results with linear pieces (i << k) + c over the source bits and emits a branch-free twiddle when few pieces suffice, a lookup otherwise. This covers all 12 exact widenings among the built-in types (and user types), replacing the hand-written Float4_E2M1FN => Float8_E4M3(FN) specializations and TWIDDLED_PAIRS. Packed conversions of any length inherit the twiddle through the lanewise path. The cutoff is max_twiddle_cost(): 8 on the host, where twiddles vectorize and beat the gather by 4-29x, and 2 on CUDA devices, where a cached constant-table load beats longer twiddles. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The piecewise fit now works on any unsigned table width, so @microfloat registers a generated cvt(F, x) per type that computes Float16, BFloat16 and Float32 results with a bit-twiddle over the destination bits. Zero and all subnormals share one head piece evaluated on the FPU (the significand under the exponent of 2^(1-bias), minus 2^(1-bias)), so every built-in type costs 3-5 instead of up to 9 linear pieces. Float64 extends the Float32 result, which beats 64-bit-lane twiddles. On the CPU this is 2-4x faster than the BFloat16 lookup for most type/destination pairs. Devices keep the lookup (or native PTX) through max_twiddle_cost, which measured faster there. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #35 +/- ##
==========================================
+ Coverage 80.49% 84.46% +3.96%
==========================================
Files 15 14 -1
Lines 482 605 +123
==========================================
+ Hits 388 511 +123
Misses 94 94 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
cvt_twiddle is the new default cvt for Float32 sources, and so for Float16, BFloat16 and every other source that converts to Float32 first. Target normals round the Float32 bits with an integer add at the target's last significand bit and rebias with a subtraction; target subnormals round |x| scaled to units of the smallest subnormal on the FPU. Overflow, NaN and sign are selects, and the constants fold from the type's layout. The error checks are the only branches and fold away for formats that cannot fail. It covers all six rounding modes and both overflow policies, and matches cvt_generic bit for bit, errors included: checked exhaustively over all 2^32 Float32 inputs for every built-in format, mode and policy. On the CPU it is ~10x faster for signed formats with NaN (the loop vectorizes) and ~2x for unsigned and FiniteOnly formats, whose error branch prevents vectorization. On the GPU it is ~3x faster than the generic path, and CUDACoreExt now falls back to it where no native instruction applies. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`any(isnan, xs)` over the lanes of a pair is not unrolled in device code: it spills the lanes to local memory and loops around the single cvt. That made native FP6/FP4 pairs ~2.7x slower than the portable twiddle on the GB10; with a `|` reduction they are ~3x faster than it. Same fix for the sign guard of ue8m0 pairs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On x86 without native BFloat16 (Julia 1.11+), inlining cvt_twiddle into T(::BFloat16) let LLVM take the sign from a `bitcast bfloat to i16` of a value it had also extended with fpext. The x86 backend then reads the low half of the extended register, which is zero, so the sign OR folded away: FiniteOnly targets lost the sign of every negative BFloat16 input (the CI failures on x64 1.11, 1.12 and pre). BFloat16 sources now widen by shifting their bits, so no bfloat arithmetic reaches the conversion. Verified under x86-64 emulation on Julia 1.11 and 1.12: every BFloat16 and Float16 bit pattern, every built-in format, mode and policy matches the generic path. Adds a regression test over all BFloat16 inputs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Faster conversions in three parts. All match
cvt_genericbit for bit:cvt_genericresults replace the hand-writtenFloat4_E2M1FN => Float8_E4M3(FN)specializations. They cover every pair, including user types.Float16,BFloat16andFloat32: generated per type the same way.Float32: a new branch-free path,cvt_twiddle, for all six rounding modes and both overflow policies.How it works
@cvt_tablealready builds the full result table of a pair fromcvt_generic. It now also fits that table with linear pieces(i << k) + cover the source bits, chosen with branch-free compares and selects:A table that fits in few pieces becomes a twiddle; otherwise it stays a lookup. Because the code is built from the reference results, it is correct by construction. All 12 exact built-in pairs twiddle at cost 1–5 (pieces, plus one for the sign move). E2M1→E2M3 is a single shift. For E2M1→E4M3FN the output reproduces the old hand-written code.
specializations.jlandTWIDDLED_PAIRSare removed. Packed conversions of any length compile to straight-line code through the lanewise path. For x2 this is identical to the old hand-written packed method.Widening:
@microfloatregisters a generatedcvt(F, x)per type, fitted over the bits ofFloat16,BFloat16orFloat32. Zero and all subnormals share one "head" piece evaluated on the FPU: put the subnormal significand under the exponent of 2^(1−bias), then subtract 2^(1−bias). The result is exact and normalized, with no subnormal operands.Float64extends theFloat32result.Per-target cutoff
New public hook
max_twiddle_cost(): 8 on the host, overridden to 2 inCUDACoreExt. Each generated method contains both forms, and the choice folds when the method is compiled.ld.global.nc) beats twiddles above cost 2The table only wins on the CPU above about 25 pieces. On the GPU, a lone shift still beats the table.
Narrowing from Float32
cvt_twiddleis the new defaultcvtforFloat32sources, and so forFloat16,BFloat16and every other source that converts throughFloat32first.Float32bits at the target's last significand bit, carrying into the exponent, and a subtraction adjusts the exponent bias.|x|is scaled to units of the smallest subnormal and rounded on the FPU.Results and errors match
cvt_genericover all 2³²Float32inputs for every built-in format × 6 modes × 2 policies. This was checked offline; skipped inputs are only those that throw in both paths. The test suite uses boundary, tie and random inputs, on the GPU as well.cvt_genericCUDACoreExtnow falls back to it where no native instruction appliesNote: for formats without significand bits (E8M0),
cvt_generic's RoundNearest rounds ties away from zero rather than to even: its odd-last-bit test reads the implicit leading 1.cvt_twiddlereproduces this exactly and documents it in a comment. Changing it would change E8M0 results, so it's left for a separate change.Tests
cvt_generic, now including Float16.The full suite passes locally, GPU tests included (50,588).
@cvt_tablekeeps its name, though it now emits twiddles as well as tables.🤖 Generated with Claude Code