Repository navigation
Upgrade to Transformers 5.17 and Hugging Face Hub 1.33 - #286
Conversation
- Pin transformers==5.17.0 and huggingface-hub==1.33.0. - Drop local_dir_use_symlinks from snapshot_download (removed in Hub 1.x). - Autospec the snapshot_download stub and check that http_get/xet_get still accept _tqdm_bar, so Hub API changes fail in the unit tests. - Isolate HF_HOME/HF_XET_CACHE in the pool download test. - Add an opt-in real CrisperWhisper test (word timestamp baseline, long-form WER, encoder attention suppression) and an e2e-crisper CI job. - Install unit-test deps with lockfile constraints in CI. - Document the checks to run before bumping either dependency.
The job resolved transitive dependencies freshly, so CI could test other versions than the ones that ship. Export constraints from uv.lock and keep the CPU torch override; the locked Linux pin torch==2.9.1 also matches the +cpu build.
Brings in #286, which already contains develop's engine split (#282). - Register Qwen3 in the engine's TRANSCRIBERS through a thin Qwen3Transcriber adapter (models still load per call; split into load/transcribe is a follow-up). - Keep TRANSFORMERS_BACKENDS in transcribe.py so Qwen3 runs in a child process and returns its transcript like CrisperWhisper. - Fold the Qwen3 language-detection e2e test into the short-clip test to save one model load per CI run.
BW-Projects
left a comment
There was a problem hiding this comment.
Looks good to me. Checked on Linux (Python 3.12, CUDA) and Windows:
- Unit and core tests pass. The pool test downloads
tinythrough Hub 1.33 with Xet and the progress dict fills, so thehttp_get/xet_getpatches still work.scripts/refresh_model_hashes.py --checkruns on 1.33 too. - Model folders come out flat with no symlinks and the hashes verify. Hub 1.x adds
.cache/huggingface/trees/and aCACHEDIR.TAG, harmless for us. - The opt-in CrisperWhisper test passes on CPU. I also ran its checks on the GPU path (CUDA, float16), which the test does not cover: the same word timestamps as the 4.57.3 baseline, 2.2 % WER on the 45 s excerpt and the encoder-attention patch still takes effect.
- Models page on Linux: tiny, base, speaker-detection and large-swedish download with working progress and verify.
- Windows (Win11 VM, frozen build from a fork run of release.yml):
basedownloads through the Models page with progress, the files land flat and match the manifest.
For #318/#320: Hub 1.x replaces requests with httpx, which ignores REQUESTS_CA_BUNDLE and reads SSL_CERT_FILE instead. I checked by pointing each variable at a self-made CA: 0.36 only fails with REQUESTS_CA_BUNDLE, 1.33 only with SSL_CERT_FILE. So once this is merged, the workaround documented in #320 no longer works unless it also sets SSL_CERT_FILE. Truststore from #319 would make both unnecessary. The connectivity check from #312 works with 1.33.
Small things, none blocking:
- pyproject.toml line 38 has three spaces of indentation.
- Hub 1.x logs "You are sending unauthenticated requests to the HF Hub..." once per process.
- crisperwhisper 2.0.3 still calls Transformers with arguments that 5.x only accepts with deprecation warnings (
torch_dtype,generation_configtogether with separate generation arguments,output_attentions). Fine with the exact pin. When a later Transformers drops them, CrisperWhisper breaks first, so that is the place to check on the next bump. - Windows now gets hf-xet: Hub 1.33's dependency marker includes
AMD64, so the lock installs it there for the first time. The frozen build ships the module but not its package metadata, and the Hub checks that metadata to decide whether Xet is available. So the MSIX still downloads over plain HTTP, there was no Xet cache in the VM after downloadingbase. Downloads work, just without Xet.copy_metadata('hf_xet')in freeze.spec should switch it on, fine as a follow-up.
# Conflicts: # pyproject.toml # uv.lock
|
Hi @BW-Projects thanks for the review and the points raised.
Now everything is ready for the qwen models in #288. PR is still in draft though, as I need to open the model repositories in our huggingface. |
as per @BW-Projects suggetion, this pr contains the transformers upgrade after positive evaluation of transformers 5.17. This enables new transformer models to be added.
So far it looks good. crisper_whisper sill works.
huggingface-hub had to be updated to 1.33.0 with all tests indicating it still working. Had to drop local_dir_use_symlinks from snapshot_download (removed in Hub 1.x).
Thus: - Pin transformers==5.17.0 and huggingface-hub==1.33.0.
Towards #280