Skip to content

Upgrade to Transformers 5.17 and Hugging Face Hub 1.33 - #286

Merged
JuergenFleiss merged 10 commits into
developfrom
feature_transformer5
Oct 29, 2026
Merged

JuergenFleiss merged 10 commits into
developfrom
feature_transformer5

Conversation

@JuergenFleiss

@JuergenFleiss JuergenFleiss commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

as per @BW-Projects suggetion, this pr contains the transformers upgrade after positive evaluation of transformers 5.17. This enables new transformer models to be added.

So far it looks good. crisper_whisper sill works.
huggingface-hub had to be updated to 1.33.0 with all tests indicating it still working. Had to drop local_dir_use_symlinks from snapshot_download (removed in Hub 1.x).

Thus: - Pin transformers==5.17.0 and huggingface-hub==1.33.0.

  • Add an opt-in real CrisperWhisper test (word timestamp baseline, long-form WER, encoder attention suppression) and an e2e-crisper CI job.

Towards #280

- Pin transformers==5.17.0 and huggingface-hub==1.33.0.
- Drop local_dir_use_symlinks from snapshot_download (removed in Hub 1.x).
- Autospec the snapshot_download stub and check that http_get/xet_get
  still accept _tqdm_bar, so Hub API changes fail in the unit tests.
- Isolate HF_HOME/HF_XET_CACHE in the pool download test.
- Add an opt-in real CrisperWhisper test (word timestamp baseline,
  long-form WER, encoder attention suppression) and an e2e-crisper CI job.
- Install unit-test deps with lockfile constraints in CI.
- Document the checks to run before bumping either dependency.
@JuergenFleiss
JuergenFleiss marked this pull request as ready for review October 2, 2026 20:56
The job resolved transitive dependencies freshly, so CI could test other
versions than the ones that ship. Export constraints from uv.lock and keep
the CPU torch override; the locked Linux pin torch==2.9.1 also matches the
+cpu build.
JuergenFleiss added a commit that referenced this pull request Oct 8, 2026
Brings in #286, which already contains develop's engine split (#282).

- Register Qwen3 in the engine's TRANSCRIBERS through a thin
  Qwen3Transcriber adapter (models still load per call; split
  into load/transcribe is a follow-up).
- Keep TRANSFORMERS_BACKENDS in transcribe.py so Qwen3 runs in a
  child process and returns its transcript like CrisperWhisper.
- Fold the Qwen3 language-detection e2e test into the short-clip
  test to save one model load per CI run.

@BW-Projects BW-Projects left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Checked on Linux (Python 3.12, CUDA) and Windows:

  • Unit and core tests pass. The pool test downloads tiny through Hub 1.33 with Xet and the progress dict fills, so the http_get/xet_get patches still work. scripts/refresh_model_hashes.py --check runs on 1.33 too.
  • Model folders come out flat with no symlinks and the hashes verify. Hub 1.x adds .cache/huggingface/trees/ and a CACHEDIR.TAG, harmless for us.
  • The opt-in CrisperWhisper test passes on CPU. I also ran its checks on the GPU path (CUDA, float16), which the test does not cover: the same word timestamps as the 4.57.3 baseline, 2.2 % WER on the 45 s excerpt and the encoder-attention patch still takes effect.
  • Models page on Linux: tiny, base, speaker-detection and large-swedish download with working progress and verify.
  • Windows (Win11 VM, frozen build from a fork run of release.yml): base downloads through the Models page with progress, the files land flat and match the manifest.

For #318/#320: Hub 1.x replaces requests with httpx, which ignores REQUESTS_CA_BUNDLE and reads SSL_CERT_FILE instead. I checked by pointing each variable at a self-made CA: 0.36 only fails with REQUESTS_CA_BUNDLE, 1.33 only with SSL_CERT_FILE. So once this is merged, the workaround documented in #320 no longer works unless it also sets SSL_CERT_FILE. Truststore from #319 would make both unnecessary. The connectivity check from #312 works with 1.33.

Small things, none blocking:

  • pyproject.toml line 38 has three spaces of indentation.
  • Hub 1.x logs "You are sending unauthenticated requests to the HF Hub..." once per process.
  • crisperwhisper 2.0.3 still calls Transformers with arguments that 5.x only accepts with deprecation warnings (torch_dtype, generation_config together with separate generation arguments, output_attentions). Fine with the exact pin. When a later Transformers drops them, CrisperWhisper breaks first, so that is the place to check on the next bump.
  • Windows now gets hf-xet: Hub 1.33's dependency marker includes AMD64, so the lock installs it there for the first time. The frozen build ships the module but not its package metadata, and the Hub checks that metadata to decide whether Xet is available. So the MSIX still downloads over plain HTTP, there was no Xet cache in the VM after downloading base. Downloads work, just without Xet. copy_metadata('hf_xet') in freeze.spec should switch it on, fine as a follow-up.

@JuergenFleiss

Copy link
Copy Markdown
Collaborator Author

Hi @BW-Projects

thanks for the review and the points raised.

Now everything is ready for the qwen models in #288. PR is still in draft though, as I need to open the model repositories in our huggingface.

@JuergenFleiss
JuergenFleiss merged commit 951068c into develop Oct 29, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants