Skip to content

feat: support all CLAP2 model sizes - #75

Merged
zhenghaoz merged 3 commits into
mainfrom
feat/clap2-all-sizes
Nov 8, 2026
Merged

zhenghaoz merged 3 commits into
mainfrom
feat/clap2-all-sizes

Conversation

@zhangzhenghao

@zhangzhenghao zhangzhenghao commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Support and document all five published CLAP2 sizes: Large (960), Medium (640), Small (512), XS (320), and XXS (224), in F32/F16.
  • Infer per-size source identity/revision instead of hardcoding Large; retain explicit revision overrides and avoid assigning Large provenance to custom geometries. The native implementation was already dimension-generic, so no inference kernel changes are needed.
  • Audit every published tensor name/shape offline; exercise all five RoPE geometries including XXS head56, and run official asset-specific numerical oracles for reduced head56 and asymmetric head64 towers.

Verification

  • Fresh Linux system-ICU static build: 36/36 default CTests.
  • Fresh Linux system-ICU shared build: 35/35 default CTests.
  • Optional official CLAP2 tests: 7/7, including all three tiny profiles, each F32/F16, with tokenizer/padding and crop-sensitive >30-second audio coverage.
  • All five published checkpoints: 20/20 native comparisons (five sizes × F32/F16 × text/audio), against pinned official open_clip and timm 1.0.30. Six diverse texts and three waveform lengths per storage mode.
  • Real XXS F16 CLI text/audio smoke: both return unit-normalized 224-dimensional vectors.
  • Independent staged-code review and scoped follow-up reviews.

Published checkpoint maximum absolute embedding errors

Size Text F32 Audio F32 Text F16 Audio F16
large 8.9407e-08 1.01961e-05 5.96046e-08 1.01961e-05
medium 7.45058e-08 1.20811e-05 8.9407e-08 1.20923e-05
small 8.9407e-08 7.33929e-06 1.19209e-07 7.36583e-06
xs 1.49012e-07 1.34697e-05 1.49012e-07 1.35042e-05
xxs 9.68575e-08 1.67266e-05 1.19209e-07 1.668e-05

Published-checkpoint tolerance remains 8e-5. The synthetic asymmetric-head64 multiple pure-tone case alone has an explicit 2e-4 waveform tolerance: F32 MKL FFT residuals near the −100 dB clamp differ from native double FFT. Independent boundary injection gave <3e-7 tower error with identical official mel, and Torch double FFT reproduced native-like differences. Text, other waveform cases and published thresholds are unchanged; negative probes confirmed their strict checks still reject altered outputs. No production FFT or inference behavior was changed.

Independent frontend/tower boundary evidence

For the unchanged asymmetric-head64 multiple input and weights:

Boundary experiment Maximum absolute embedding error
Native complete waveform path 1.0035932e-4
Official Torch mel → native patchify/audio tower 2.83122e-7
Native power spectrum + Torch mel filters 1.0034442e-4
Torch F32 power spectrum + native mel filters 3.27826e-7

The worst near-floor mel bin differed by 0.28366 dB (native −99.24464, Torch −99.52831). Exact Torch Hann with native double FFT still showed the discrepancy; official double FFT reproduced native-like results. This evidence was independently reviewed using retained raw mel artifacts. It is not presented as a production inference change.

CLAP2 block quantization is not added by this change. Cross-platform CI status is reported separately from the local/published checks above.

Remote CI

Commit 8f0e96680c61c96edc9c2a5493fe88368c0af1f0, run 42484177700: 9/9 checks passed.

  • Native build/tests/install: Linux x64/ARM64, macOS ARM64, Windows x64/ARM64.
  • Python/Node package tests: Linux x64/ARM64, macOS ARM64, Windows x64. Windows ARM64 intentionally skips these binding jobs in the existing workflow.
  • Rust: Linux, macOS, Windows.
  • clang-format.

@zhenghaoz
zhenghaoz merged commit 60def51 into main Nov 8, 2026
9 checks passed
@zhenghaoz
zhenghaoz deleted the feat/clap2-all-sizes branch November 8, 2026 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants