[v0.2.5+] Support Types #58

Activity

1. m96-chan commented on Dec 14, 2025

   

   m96-chan

   on Dec 14, 2025

   OwnerAuthor

   More actions

   FP16/BF16 Support Implementation Complete

   FP16 (float16) and BF16 (bfloat16) support has been implemented for all core operators.

   Implementation Summary

   C++ Native Backend:

   • Added Float16 and BFloat16 to DataType enum in native/core/types.hpp
   • Implemented FP16/BF16 kernels for all elementwise operations (add, sub, mul, div)
   • Implemented FP16/BF16 kernels for all unary operations (exp, log, relu)
   • Implemented FP16/BF16 kernels for all reduction operations (sum, mean, max)
   • Implemented FP16/BF16 matmul in native/ops/matmul_f16_bf16.cuh
   • All operations use FP32 accumulation for numerical stability

   Python API:

   • Added float16 and bfloat16 DataType objects in pygpukit.core.dtypes
   • Updated GPUArray._wrap_native() to handle Float16/BFloat16
   • Updated validation functions to allow FP16/BF16 for float operations
   • Exported float16 and bfloat16 in pygpukit.__init__

   Pybind11 Bindings:

   • Added Float16/BFloat16 to DataType enum export
   • Updated from_numpy() to detect np.float16 and np.uint16 (for BF16)
   • Updated to_numpy() to create appropriate output arrays

   Updated Status Table

   Operation	FP32	TF32	FP16	BF16
   add	✅	—	✅	✅
   sub	✅	—	✅	✅
   mul	✅	—	✅	✅
   div	✅	—	✅	✅
   exp	✅	—	✅	✅
   log	✅	—	✅	✅
   relu	✅	—	✅	✅
   sum	✅	—	✅	✅
   mean	✅	—	✅	✅
   max	✅	—	✅	✅
   matmul	✅ (18 TFLOPS)	✅ (27.38 TFLOPS)	✅	✅

   Test Results

   All 166 existing tests pass. Comprehensive FP16/BF16 tests (22 additional tests) all pass:

   • FP16: add, sub, mul, div, exp, log, relu, sum, mean, max, matmul
   • BF16: add, sub, mul, div, exp, log, relu, sum, mean, max, matmul

   Notes

   • FP16/BF16 matmul uses simple kernels with FP32 accumulation for correctness
   • TensorCore WMMA optimization for FP16/BF16 matmul can be added in future for better performance
   • BF16 uses uint16 storage in NumPy (no native bfloat16 type)
2. m96-chan

   added 3 commits that reference this issue on Dec 15, 2025

   wip(#58): add FP16/BF16 support and reduction ops

   ...

   990a691

   feat(#58): add operator overloads and astype method to GPUArray

   ...

   0264538

   v0.2.5: FP16/BF16 support, TF32 optimization, reduction ops (#61)

   ...

   854719b
3. m96-chan commented on Dec 15, 2025

   

   m96-chan

   on Dec 15, 2025

   OwnerAuthor

   More actions

   Completed in v0.2.5

   FP16/BF16 Data Types

   • Added gpk.float16 and gpk.bfloat16 dtypes
   • FP32 accumulation for numerical stability
   • astype() method for dtype conversion

   Elementwise Operations

   All elementwise ops now support FP16/BF16:

   • add, sub, mul, div
   • Operator overloads: +, -, *, /

   Matrix Multiplication

   • FP16 matmul with FP32 accumulation
   • BF16 matmul with FP32 accumulation

   Benchmark Results (RTX 3090 Ti)

   Size	FP16	BF16
   2048²	2.4 TFLOPS	2.4 TFLOPS
   4096²	2.4 TFLOPS	2.3 TFLOPS
   8192²	2.3 TFLOPS	2.3 TFLOPS

   Note: TensorCore optimization for FP16/BF16 tracked in #60

   PR: #61
4. m96-chan

   closed this as completedon Dec 15, 2025