Skip to content

test: unify architecture-specific model contracts - #77

Open
zhangzhenghao wants to merge 1 commit into
mainfrom
refactor/unified-model-tests
Open

zhangzhenghao wants to merge 1 commit into
mainfrom
refactor/unified-model-tests

Conversation

@zhangzhenghao

@zhangzhenghao zhangzhenghao commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Follow llama.cpp's architecture-selected test runner and per-model tokenizer registration pattern: one shared runner, with architecture fixtures and expectations registered separately.

  • Introduce tests/models/model_tests.py with --arch, --suite, --case and fixture-free --list.
  • Centralize subprocess execution, exit/diagnostic assertions, JSON vector/frame comparisons, WAV generation and synthetic GGUF metadata rewriting in model_contract.py.
  • Replace six standalone CLAP/CLAP2/CLaMP3/MERT/MERT2 scripts with architecture adapters; this is not a dispatcher to the old scripts.
  • Register existing CTest cases through one helper, preserving names, option gates, dependencies, fixture setup and timeouts.
  • Retain every migrated mutation and numerical threshold, including positive-only MERT rejection exits, strict CLAP error bounds, inclusive MERT2 bounds and its 120-second invocation deadlines.
  • Keep specialized official numerical oracles, native C API tests, FFT/RoPE/tokenizer internals, bundled pipelines and MuQ-specific contracts separate and unchanged.
  • Add 21 framework regressions and document adding/selecting architecture contracts.

Reference: https://github.com/ggml-org/llama.cpp/blob/master/tests/test-llama-archs.cpp and https://github.com/ggml-org/llama.cpp/blob/master/tests/CMakeLists.txt.

Verification

  • Default static Release build: 36/36 CTests passed.
  • Default shared Release build: 35/35 CTests passed (allocation-interposition test remains static-only).
  • Optional CLAP/CLaMP3/MERT2 parity, native, tokenizer and migrated CLI/metadata contracts with freshly generated fixtures: 10/10 CTests passed.
  • Framework regressions: 21 passed.
  • Deliberately corrupted a MERT2 pooled reference: real CLI succeeded, unified test correctly exited 1 with architecture/suite/case, numerical error and captured command output.
  • Independent semantic review: no blocking findings; MERT base config and all 31 native mutations mechanically compared with the baseline.
  • Python compilation and git diff --check passed.

An initial run against historical CLaMP3 fixtures failed because those GGUFs lacked bundled MERT metadata; the baseline script fails identically. Regenerating fixtures with the current parity pipeline passed all optional checks.

GitHub Actions on b1c62939daf9b4f05850553e1c6753cc0754f1e5: all 9 checks passed — clang-format; Linux x64/ARM64, macOS ARM64 and Windows x64/ARM64 builds/tests; Rust checks on Linux/macOS/Windows. Verified CI run: https://github.com/gorse-io/clap.cpp/actions/runs/42498659326.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants