diff --git a/README.md b/README.md index 3ad96b1..b1c0c27 100644 --- a/README.md +++ b/README.md @@ -559,4 +559,4 @@ Ettore Di Giacinto ([@mudler](https://github.com/mudler)). ## License -parakeet.cpp is released under the [MIT License](LICENSE). The model weights are governed by the licenses of the original models, so check each model card on HuggingFace. The NVIDIA Parakeet models are mostly CC-BY-4.0 (nemotron-3.5-asr-streaming is OpenMDW-1.1). Moondream's [parakeet-ultra](https://huggingface.co/moondream/parakeet-ultra) and [parakeet-redux](https://huggingface.co/moondream/parakeet-redux) are CC-BY-4.0 too: credit Moondream and NVIDIA ([parakeet-tdt-0.6b-v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3)), link the [license](https://creativecommons.org/licenses/by/4.0/), and note that GGUF files made here are converted (and quantized, or dequantized for Redux) copies, not retrained models. +parakeet.cpp is released under the [MIT License](LICENSE). The model weights are governed by the licenses of the original models, so check each model card on HuggingFace. The NVIDIA Parakeet models are mostly CC-BY-4.0 (nemotron-3.5-asr-streaming and Nemotron-3-Diarization are OpenMDW-1.1, parakeet_realtime_eou_120m-v1 is the NVIDIA Open Model License, and the Silero VAD files are MIT). See [docs/licenses.md](docs/licenses.md) for the full table. Moondream's [parakeet-ultra](https://huggingface.co/moondream/parakeet-ultra) and [parakeet-redux](https://huggingface.co/moondream/parakeet-redux) are CC-BY-4.0 too: credit Moondream and NVIDIA ([parakeet-tdt-0.6b-v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3)), link the [license](https://creativecommons.org/licenses/by/4.0/), and note that GGUF files made here are converted (and quantized, or dequantized for Redux) copies, not retrained models. diff --git a/docs/licenses.md b/docs/licenses.md new file mode 100644 index 0000000..b24dbfb --- /dev/null +++ b/docs/licenses.md @@ -0,0 +1,38 @@ +# Model licences + +The GGUF files published at `mudler/parakeet-cpp-gguf` are converted from other +people's models, so each file family keeps the licence of its source model. The +parakeet.cpp code is MIT-licensed. + +This table is the reference for `scripts/publish_hf.py`. The script has no default +licence: a model id that is missing from its `LICENSES` table is an error. The +check `tests/python/check_publish_hf_licences.py` (and +`python3 scripts/publish_hf.py --check-licences`) fails when this table and the +script differ, so change both together. + +| Source checkpoint | Licence | Licence text | Notice to keep | +|---|---|---|---| +| `moondream/parakeet-redux` | CC-BY-4.0 | | | +| `moondream/parakeet-ultra` | CC-BY-4.0 | | | +| `nvidia/Nemotron-3-Diarization` | OpenMDW 1.1 | | | +| `nvidia/nemotron-3.5-asr-streaming-0.6b` | OpenMDW 1.1 | | | +| `nvidia/parakeet-ctc-0.6b` | CC-BY-4.0 | | | +| `nvidia/parakeet-ctc-1.1b` | CC-BY-4.0 | | | +| `nvidia/parakeet-rnnt-0.6b` | CC-BY-4.0 | | | +| `nvidia/parakeet-rnnt-1.1b` | CC-BY-4.0 | | | +| `nvidia/parakeet-tdt-0.6b-v2` | CC-BY-4.0 | | | +| `nvidia/parakeet-tdt-0.6b-v3` | CC-BY-4.0 | | | +| `nvidia/parakeet-tdt-1.1b` | CC-BY-4.0 | | | +| `nvidia/parakeet-tdt_ctc-1.1b` | CC-BY-4.0 | | | +| `nvidia/parakeet-tdt_ctc-110m` | CC-BY-4.0 | | | +| `nvidia/parakeet_realtime_eou_120m-v1` | NVIDIA Open Model License | | Licensed by NVIDIA Corporation under the NVIDIA Open Model License | +| `snakers4/silero-vad` | MIT | | Copyright (c) 2020-present Silero Team. Released under the MIT license | + +Notes: + +- The Moondream Ultra and Redux files are converted here, not trained. Credit + Moondream and NVIDIA (`parakeet-tdt-0.6b-v3`) when you use them. The Redux F16 and + Q8_0 files are dequantized from the ternary weights. +- The Silero VAD files are converted from the official ONNX model, not trained. +- Static card text for the files that `publish_hf.py` does not convert (diarization, + Moondream, Silero VAD, and the Nemotron 3.5 usage note) lives in `scripts/hf_card/`. diff --git a/scripts/hf_card/diarization.md b/scripts/hf_card/diarization.md new file mode 100644 index 0000000..ef30eaf --- /dev/null +++ b/scripts/hf_card/diarization.md @@ -0,0 +1,14 @@ +### nemotron-3-diarization + +Source: [nvidia/Nemotron-3-Diarization](https://huggingface.co/nvidia/Nemotron-3-Diarization) · Speaker diarization (Sortformer, up to 8 speakers, streaming speaker cache) · License: [OpenMDW 1.1](https://openmdw.ai/license/1-1/) + +| File | Variant | Size | Segments vs NeMo | +|---|---|---:|---:| +| `nemotron-3-diarization-f16.gguf` ← **recommended** | F16 | 200.7 MB | identical | +| `nemotron-3-diarization-q8_0.gguf` | Q8_0 | 108.7 MB | identical on 2 of 3 clips, 99.5% of frames on the third | + +> Checked against NeMo on a 23.6 s and a 68.5 s two-speaker clip (offline and streaming: every segment identical, to the 10 ms frame) and a 12.3 min three-speaker clip (F16 100%, Q8_0 99.9% of speech frames). Q8_0 can flip a frame whose probability sits right at the 0.5 threshold, splitting a segment (99.5% of frames on a 31.5 s clip); use F16 when segment-exact output matters. Answers "who spoke when"; pair it with any ASR model above for speaker-attributed transcripts. See [diarization.md](https://github.com/mudler/parakeet.cpp/blob/master/docs/diarization.md). + +```bash +build/examples/cli/diarize models/nemotron-3-diarization-f16.gguf meeting.wav +``` diff --git a/scripts/hf_card/moondream.md b/scripts/hf_card/moondream.md new file mode 100644 index 0000000..8c697ca --- /dev/null +++ b/scripts/hf_card/moondream.md @@ -0,0 +1,25 @@ +### ultra and redux (Moondream) + +Source: [moondream/parakeet-ultra](https://huggingface.co/moondream/parakeet-ultra) and [moondream/parakeet-redux](https://huggingface.co/moondream/parakeet-redux) by [Moondream](https://huggingface.co/moondream), derived from NVIDIA's [nvidia/parakeet-tdt-0.6b-v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) · TDT transducer (FastConformer), multilingual, with a voice-activity head (`transcribe --vad`) · heads: TDT · license: [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) + +| File | Variant | Size | Runs on | +|---|---|---:|---| +| `ultra-f16.gguf` | F16 | 1441.9 MB | any backend | +| `ultra-q8_0.gguf` | Q8_0 | 941.5 MB | any backend | +| `redux-packed.gguf` | packed ternary encoder | 213.3 MB | CPU only, offline only | +| `redux-f16.gguf` | F16, dequantized | 1441.9 MB | any backend | +| `redux-q8_0.gguf` | Q8_0, dequantized | 941.5 MB | any backend | + +> There is no NeMo reference for these models, so there is no WER column. Each file was checked to decode the `tests/fixtures/speech.wav` clip to the expected sentence. See [ternary.md](https://github.com/mudler/parakeet.cpp/blob/master/docs/ternary.md) for the measurements. + +- **Ultra** has ordinary F16 weights and runs on any backend, like the other v3 files. +- **Redux packed** keeps the encoder linear layers as ternary weights (-1, 0 or +1 times a per-group scale) and runs a native CPU kernel. It does not run on GPU backends or in streaming mode, and parakeet.cpp refuses to load it there. For GPU use, take `redux-f16.gguf` or `redux-q8_0.gguf`. +- **Redux F16 and Q8_0** are dequantized: the ternary weights were expanded to ordinary weights and then stored as F16 or Q8_0. They are not packed. +- **Changes:** these files are converted here, not trained. Nothing was trained or fine-tuned by the parakeet.cpp project. The models were trained by NVIDIA (the base) and Moondream (Ultra and Redux). + +```bash +huggingface-cli download mudler/parakeet-cpp-gguf redux-packed.gguf --local-dir models/ +build/examples/cli/parakeet-cli transcribe --model models/redux-packed.gguf --input audio.wav +# Long audio: cut at pauses with the model's voice-activity head (offline only). +build/examples/cli/parakeet-cli transcribe --model models/ultra-q8_0.gguf --input long.wav --vad +``` diff --git a/scripts/hf_card/nemotron-3.5-note.md b/scripts/hf_card/nemotron-3.5-note.md new file mode 100644 index 0000000..13eec2e --- /dev/null +++ b/scripts/hf_card/nemotron-3.5-note.md @@ -0,0 +1,6 @@ +> This model runs both offline and as a cache-aware streaming model, and it is the only one here that takes a target language. Pass `--lang ` (for example `en-US`, `de-DE`, `es-ES`, `ja-JP`), or leave it at the default `auto` to let the model detect the language. The WER column is measured against NeMo, offline, on the languages en, de and auto; streaming and offline output also match NeMo for en, de, es, ja-JP and auto. See [parity.md](https://github.com/mudler/parakeet.cpp/blob/master/docs/parity.md). The small prompt layers, the LSTM and the feature extractor stay F32 in every quantization. + +```bash +huggingface-cli download mudler/parakeet-cpp-gguf nemotron-3.5-asr-streaming-0.6b-f16.gguf --local-dir models/ +build/examples/cli/parakeet-cli transcribe --model models/nemotron-3.5-asr-streaming-0.6b-f16.gguf --input audio.wav --lang de-DE +``` diff --git a/scripts/hf_card/silero-vad.md b/scripts/hf_card/silero-vad.md new file mode 100644 index 0000000..cfa5462 --- /dev/null +++ b/scripts/hf_card/silero-vad.md @@ -0,0 +1,13 @@ +### silero-vad (Silero Team) + +Source: [snakers4/silero-vad](https://github.com/snakers4/silero-vad) v6.2.3 by the [Silero Team](https://github.com/snakers4) · a small voice-activity detector, not a speech recognizer · 8 kHz and 16 kHz in one file · license: [MIT](https://github.com/snakers4/silero-vad/blob/master/LICENSE) + +| File | Variant | Size | +|---|---|---:| +| `silero-vad-f32.gguf` | F32 | 2.2 MB | +| `silero-vad-f16.gguf` | F16 weights, widened to F32 at load | 1.3 MB | + +- The files are converted from the official ONNX model with `scripts/convert_silero_vad_to_gguf.py` in parakeet.cpp. The GGUF records the source version and the ONNX sha256. They were converted here, not trained. +- They need a parakeet.cpp build that has the standalone VAD API. That code is not in a release yet, so check the parakeet.cpp repository before relying on it. +- Use it as a stand-alone detector (`parakeet-cli vad --model silero-vad-f16.gguf --input audio.wav`) or to cut long audio before transcribing with any model, including those that have no VAD head (`parakeet-cli transcribe --model tdt-0.6b-v3-q8_0.gguf --input long.wav --vad --vad-model silero-vad-f16.gguf`). +- Probabilities match onnxruntime to about 1e-6 (F32) and 1e-3 (F16) on the test clips. diff --git a/scripts/publish_hf.py b/scripts/publish_hf.py index bc5eaf0..671c851 100644 --- a/scripts/publish_hf.py +++ b/scripts/publish_hf.py @@ -121,33 +121,134 @@ "q8_0": {"wer": 0.0, "size_mb": None}, "q4_k": {"wer": None, "size_mb": None}, }, - # Multilingual prompt-conditioned streaming model. WER measured offline - # (en/de/auto) by scripts/e2e_nemo_compare.py vs NeMo; all five variants are - # byte-for-byte (WER 0.0). prompt_kernel + LSTM + featurizer tensors stay F32. + # Multilingual prompt-conditioned streaming model. The WER is agreement with + # NeMo, not accuracy on a speech corpus. Recorded evidence (see docs/parity.md, + # scripts/e2e_nemo_compare.py, benchmarks/results/nemotron/bench.json): + # * F32: 20 rows (speech.wav + clip.wav x en/de/es/ja-JP/auto x offline/stream), WER 0. + # * Q8_0: speech.wav, en, offline, WER 0 (bench.json). + # * F16: stated in docs/parity.md, no run artifact is kept in the repo. + # * Q6_K, Q5_K, Q4_K: no measurement is recorded anywhere, so no WER is claimed. + # prompt_kernel + LSTM + featurizer tensors stay F32. "nvidia/nemotron-3.5-asr-streaming-0.6b": { "f16": {"wer": 0.0, "size_mb": 1484.3}, "q8_0": {"wer": 0.0, "size_mb": 983.7}, - "q6_k": {"wer": 0.0, "size_mb": 855.7}, - "q5_k": {"wer": 0.0, "size_mb": 784.8}, - "q4_k": {"wer": 0.0, "size_mb": 718.1}, + "q6_k": {"wer": None, "size_mb": 855.7}, + "q5_k": {"wer": None, "size_mb": 784.8}, + "q4_k": {"wer": None, "size_mb": 718.1}, }, } -# Per-model SPDX license id + human label, used in the generated card frontmatter -# and the License section. Defaults to CC-BY-4.0 (the NVIDIA NeMo Parakeet -# checkpoints); nemotron-3.5-asr-streaming is released under OpenMDW-1.1. -DEFAULT_LICENSE = ("cc-by-4.0", "CC-BY-4.0", - "https://creativecommons.org/licenses/by/4.0/") +# Per-model licence: (frontmatter license id, human label, licence url). +# +# Every model id that the script can publish MUST have an explicit entry here. +# There is no default: an unknown id raises, so a new model can never silently +# ship under the wrong licence. `--check-licences` and +# tests/python/check_publish_hf_licences.py enforce this, and also check that +# this table equals the table in docs/licenses.md. +CC_BY_4_0 = ("cc-by-4.0", "CC-BY-4.0", "https://creativecommons.org/licenses/by/4.0/") +OPENMDW_1_1 = ("other", "OpenMDW 1.1", "https://openmdw.ai/license/1-1/") +NVIDIA_OML = ( + "other", "NVIDIA Open Model License", + "https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/", +) +MIT_SILERO = ("mit", "MIT", "https://github.com/snakers4/silero-vad/blob/master/LICENSE") + LICENSES: dict = { - "nvidia/nemotron-3.5-asr-streaming-0.6b": ( - "other", "OpenMDW-1.1", "https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b", - ), + "nvidia/parakeet-tdt_ctc-110m": CC_BY_4_0, + "nvidia/parakeet_realtime_eou_120m-v1": NVIDIA_OML, + "nvidia/nemotron-3.5-asr-streaming-0.6b": OPENMDW_1_1, + "nvidia/parakeet-ctc-0.6b": CC_BY_4_0, + "nvidia/parakeet-rnnt-0.6b": CC_BY_4_0, + "nvidia/parakeet-tdt-0.6b-v2": CC_BY_4_0, + "nvidia/parakeet-tdt-0.6b-v3": CC_BY_4_0, + "nvidia/parakeet-ctc-1.1b": CC_BY_4_0, + "nvidia/parakeet-rnnt-1.1b": CC_BY_4_0, + "nvidia/parakeet-tdt-1.1b": CC_BY_4_0, + "nvidia/parakeet-tdt_ctc-1.1b": CC_BY_4_0, + # Published by other tooling, but they have sections on the same card. + "nvidia/Nemotron-3-Diarization": OPENMDW_1_1, + "moondream/parakeet-ultra": CC_BY_4_0, + "moondream/parakeet-redux": CC_BY_4_0, + "snakers4/silero-vad": MIT_SILERO, +} + +# Text the licence requires the redistributor to keep, shown on the card. +LICENSE_NOTICES: dict = { + "nvidia/parakeet_realtime_eou_120m-v1": + "Licensed by NVIDIA Corporation under the NVIDIA Open Model License", + "snakers4/silero-vad": + "Copyright (c) 2020-present Silero Team. Released under the MIT license", +} + +# Models and files on the card that this script does not convert. Their card +# sections are static text in scripts/hf_card/, so a regenerated card keeps them. +# Each entry is (model id or ids, section file). +EXTERNAL_SECTIONS: list = [ + (("nvidia/Nemotron-3-Diarization",), "diarization.md"), + (("moondream/parakeet-ultra", "moondream/parakeet-redux"), "moondream.md"), + (("snakers4/silero-vad",), "silero-vad.md"), +] +# Extra text that follows the table of a converted model (note + usage). +MODEL_EXTRA_SECTIONS: dict = { + "nvidia/nemotron-3.5-asr-streaming-0.6b": "nemotron-3.5-note.md", } +CARD_SECTIONS_DIR = SCRIPTS_DIR / "hf_card" def _license(model_id: str) -> tuple[str, str, str]: - """(frontmatter license id, human label, upstream/license url) for *model_id*.""" - return LICENSES.get(model_id, DEFAULT_LICENSE) + """(frontmatter license id, human label, licence url) for *model_id*. + + Raises KeyError for an id with no explicit entry: there is no default. + """ + try: + return LICENSES[model_id] + except KeyError: + raise KeyError( + f"no explicit licence for {model_id!r}: add it to LICENSES in " + "scripts/publish_hf.py and to docs/licenses.md" + ) from None + + +def _all_known_ids() -> List[str]: + ids = list(ALL_MODELS) + list(DERIVED_MODELS) + for group, _f in EXTERNAL_SECTIONS: + ids += list(group) + return sorted(set(ids)) + + +def check_licences() -> List[str]: + """Return a list of problems with the licence table (empty = fine).""" + problems: List[str] = [] + for m in _all_known_ids(): + if m not in LICENSES: + problems.append(f"{m}: no explicit licence entry") + for m in LICENSES: + if m not in _all_known_ids(): + problems.append(f"{m}: licence entry for an id the script does not know") + for m in DERIVED_MODELS: + base = DERIVED_MODELS[m]["base"] + if m in LICENSES and base in LICENSES and LICENSES[m] != LICENSES[base]: + problems.append(f"{m}: licence differs from its base {base}") + for m in LICENSE_NOTICES: + if m not in LICENSES: + problems.append(f"{m}: notice for an id with no licence") + if "nvidia/parakeet_realtime_eou_120m-v1" in LICENSES and \ + "Licensed by NVIDIA Corporation under the NVIDIA Open Model License" \ + not in LICENSE_NOTICES.get("nvidia/parakeet_realtime_eou_120m-v1", ""): + problems.append("realtime_eou_120m-v1: missing the required NVIDIA notice") + files = [f for _g, f in EXTERNAL_SECTIONS] + list(MODEL_EXTRA_SECTIONS.values()) + for f in files: + if not (CARD_SECTIONS_DIR / f).is_file(): + problems.append(f"missing card section file scripts/hf_card/{f}") + return problems + + +def _section_text(name: str) -> str: + return (CARD_SECTIONS_DIR / name).read_text(encoding="utf-8").rstrip("\n") + + +def _file_glob(model_id: str) -> str: + return f"{_model_slug(model_id).lower()}-*" # Moondream's derivatives of NVIDIA parakeet-tdt-0.6b-v3. They are trained by @@ -542,6 +643,9 @@ def build_model_card( f"released under the [{lic_label}]({lic_url}) license. " "The parakeet.cpp runtime is MIT-licensed." ) + if model_id in LICENSE_NOTICES: + lines.append("") + lines.append(f"Notice: {LICENSE_NOTICES[model_id]}.") lines.append("") return "\n".join(lines) @@ -649,44 +753,87 @@ def publish_model( # Single-repo collection: one repo, all models × variants as flat GGUFs # --------------------------------------------------------------------------- +def _license_section_lines(models: List[str]) -> List[str]: + """The card's `## License` section: one bullet per file family, then credits.""" + all_ids = list(models) + for group, _f in EXTERNAL_SECTIONS: + all_ids += [m for m in group if m not in all_ids] + + # Group file families by licence, in first-seen order. + groups: dict = {} + for m in all_ids: + groups.setdefault(_license(m), []).append(m) + + lines = [ + "## License", + "", + "Licences differ by model, so the front matter says `license: other`. Each file " + "family follows the licence of the model it was converted from. The files are " + "converted, not trained here. The parakeet.cpp runtime itself is MIT-licensed.", + "", + ] + for (_id, label, url), ids in groups.items(): + globs = ", ".join(f"`{_file_glob(m)}`" for m in ids) + lines.append(f"- {globs}: [{label}]({url}).") + lines.append("") + + notices = [f"{LICENSE_NOTICES[m]}" for m in all_ids if m in LICENSE_NOTICES] + if notices: + lines.append("Notices that the licences require:") + lines.append("") + for m in all_ids: + if m in LICENSE_NOTICES: + lines.append(f"- `{_file_glob(m)}`: {LICENSE_NOTICES[m]}.") + lines.append("") + + derived = [m for m in all_ids if m in DERIVED_MODELS] + if derived: + bases = sorted({DERIVED_MODELS[m]["base"] for m in derived}) + lines.append( + "`ultra-*.gguf` and `redux-*.gguf` are converted from " + "[moondream/parakeet-ultra](https://huggingface.co/moondream/parakeet-ultra) and " + "[moondream/parakeet-redux](https://huggingface.co/moondream/parakeet-redux) by " + "Moondream, which are derived from NVIDIA's " + + ", ".join(f"[{b.split('/')[-1]}](https://huggingface.co/{b})" for b in bases) + + ". Credit Moondream and NVIDIA when you use these files. They were converted " + "here, not trained, and the Redux F16 and Q8_0 files are dequantized from the " + "ternary weights." + ) + lines.append("") + return lines + + def build_collection_card( models: List[str], variants: List[str], paths_by_model: dict[str, dict[str, Path]], repo_id: str, ) -> str: - """One combined model card for the whole collection repo.""" + """One combined model card for the whole collection repo. + + *models* are the models this script converted. Files published by other + tooling (diarization, Moondream, Silero VAD) get their sections from + scripts/hf_card/ so a regeneration keeps them. + """ lines: List[str] = [] - # YAML frontmatter — base_model accepts a list. Licenses can differ per model - # (most are CC-BY-4.0; nemotron is OpenMDW-1.1), so when the collection mixes - # licenses we declare `other` here and spell each one out in the License - # section / per-model rows below. - licenses = {_license(m) for m in models} - if len(licenses) == 1: - only_id, only_label, only_url = next(iter(licenses)) - lines = ["---", f"license: {only_id}"] - if only_id == "other": - lines += [f"license_name: {only_label}", f"license_link: {only_url}"] - lines += ["library_name: parakeet.cpp", "tags:"] - else: - lines = [ - "---", "license: other", - "license_name: mixed (see per-model licenses below)", - f"license_link: https://huggingface.co/{repo_id}", - "library_name: parakeet.cpp", "tags:", - ] + # YAML frontmatter. The card always mixes licences, so it declares `other` + # and spells each licence out in the License section. + lines = [ + "---", "license: other", + "license_name: mixed-per-model-see-license-section", + f"license_link: https://huggingface.co/{repo_id}#license", + "library_name: parakeet.cpp", "tags:", + ] lines += [f" - {t}" for t in ( "automatic-speech-recognition", "asr", "parakeet", - "gguf", "ggml", "cpp-inference", "nemo", + "gguf", "ggml", "cpp-inference", "nemo", "speaker-diarization", )] lines.append("pipeline_tag: automatic-speech-recognition") lines.append("base_model:") bases = list(models) - for m in models: - b = DERIVED_MODELS.get(m, {}).get("base") - if b and b not in bases: - bases.append(b) + for group, _f in EXTERNAL_SECTIONS: + bases += [m for m in group if m not in bases] lines += [f" - {m}" for m in bases] lines += ["---", ""] @@ -705,10 +852,28 @@ def build_collection_card( ) lines.append("") - # Per-model sections with a variant table each. + # Per-model sections with a variant table each. Models with their own extra + # text (nemotron) come after the shared WER note, as on the published card. lines.append("## Models") lines.append("") - for model_id in models: + + def _wer_note() -> None: + lines.append( + "> WER (word error rate) is computed against the upstream NeMo reference on " + "`tests/fixtures/speech.wav` (LibriSpeech `2086-149220-0033`, ~7.4 s, English). " + "0.0 = byte-for-byte identical transcript. " + "See [parity.md](https://github.com/mudler/parakeet.cpp/blob/master/docs/parity.md) " + "and [quantization.md](https://github.com/mudler/parakeet.cpp/blob/master/docs/quantization.md)." + ) + lines.append("") + + ordered = [m for m in models if m not in MODEL_EXTRA_SECTIONS] + \ + [m for m in models if m in MODEL_EXTRA_SECTIONS] + note_written = False + for model_id in ordered: + if model_id in MODEL_EXTRA_SECTIONS and not note_written: + _wer_note() + note_written = True slug = _model_slug(model_id) arch_desc, heads = _arch_info(model_id) wer_data = KNOWN_WER.get(model_id, {}) @@ -721,13 +886,8 @@ def build_collection_card( f"{arch_desc} · heads: {heads} · license: [{_lic_label}]({_lic_url})" ) lines.append("") - if model_id in DERIVED_MODELS: - d = DERIVED_MODELS[model_id] - lines.append( - f"By [Moondream](https://huggingface.co/moondream), derived from NVIDIA's " - f"[{d['base']}](https://huggingface.co/{d['base']}). Converted, not trained here. " - f"{d['conversion']}" - ) + if model_id in LICENSE_NOTICES: + lines.append(f"Notice: {LICENSE_NOTICES[model_id]}.") lines.append("") lines.append("| File | Variant | Size | WER vs NeMo |") lines.append("|---|---|---:|---:|") @@ -740,15 +900,15 @@ def build_collection_card( rec = " ← **recommended**" if v == "f16" else "" lines.append(f"| `{fname}`{rec} | {v.upper()} | {size} | {wer} |") lines.append("") + if model_id in MODEL_EXTRA_SECTIONS: + lines.append(_section_text(MODEL_EXTRA_SECTIONS[model_id])) + lines.append("") + if not note_written: + _wer_note() - lines.append( - "> WER (word error rate) is computed against the upstream NeMo reference on " - "`tests/fixtures/speech.wav` (LibriSpeech `2086-149220-0033`, ~7.4 s, English). " - "0.0 = byte-for-byte identical transcript. " - "See [parity.md](https://github.com/mudler/parakeet.cpp/blob/main/docs/parity.md) " - "and [quantization.md](https://github.com/mudler/parakeet.cpp/blob/main/docs/quantization.md)." - ) - lines.append("") + for _group, fname in EXTERNAL_SECTIONS: + lines.append(_section_text(fname)) + lines.append("") # Quantization notes (shared). lines.append("## Quantization notes") @@ -782,29 +942,7 @@ def build_collection_card( lines.append("```") lines.append("") - # License. Each GGUF inherits the license of its source checkpoint; list them - # explicitly because the collection can mix licenses (most CC-BY-4.0, nemotron - # OpenMDW-1.1). - lines.append("## License") - lines.append("") - lines.append( - "Each GGUF is derived from its upstream checkpoint and inherits that checkpoint's " - "license. The files are converted, not trained here. The parakeet.cpp runtime itself " - "is MIT-licensed." - ) - lines.append("") - lines.append("| Source checkpoint | License |") - lines.append("|---|---|") - for model_id in models: - _lic_id, _lic_label, _lic_url = _license(model_id) - credit = "" - if model_id in DERIVED_MODELS: - b = DERIVED_MODELS[model_id]["base"] - credit = f" (Moondream, derived from NVIDIA [{b}](https://huggingface.co/{b}))" - lines.append( - f"| [{model_id}](https://huggingface.co/{model_id}){credit} | [{_lic_label}]({_lic_url}) |" - ) - lines.append("") + lines += _license_section_lines(models) return "\n".join(lines) @@ -1009,8 +1147,26 @@ def main() -> int: "resumable — pair with a watchdog that restarts on stall." ), ) + parser.add_argument( + "--check-licences", + action="store_true", + help=( + "Check that every model id the script can publish has an explicit " + "licence entry, print the table and exit (0 = fine). No conversion, " + "no HuggingFace calls." + ), + ) args = parser.parse_args() + if args.check_licences: + problems = check_licences() + for m in _all_known_ids(): + lic = LICENSES.get(m) + print(f"{m}: {lic[1] if lic else 'MISSING'}") + for p in problems: + print(f"error: {p}", file=sys.stderr) + return 1 if problems else 0 + # Resolve the model list ('all' → canonical set), de-duplicated, order-stable. raw_models = args.model or [] models: List[str] = [] @@ -1019,6 +1175,15 @@ def main() -> int: if one not in models: models.append(one) + unlicensed = [m for m in models if m not in LICENSES] + if unlicensed: + print( + "error: no explicit licence for: " + ", ".join(unlicensed) + ". Add each id to " + "LICENSES in scripts/publish_hf.py (and docs/licenses.md) before publishing.", + file=sys.stderr, + ) + return 2 + variants: List[str] = [v.strip() for v in args.variants.split(",") if v.strip()] output_dir = Path(args.output_dir) upload: bool = args.upload diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 0c7342b..d922d6c 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -221,6 +221,9 @@ if(PARAKEET_PY) add_test(NAME check_hf_convert COMMAND ${PARAKEET_PY} ${CMAKE_SOURCE_DIR}/tests/python/check_hf_convert.py) set_tests_properties(check_hf_convert PROPERTIES SKIP_RETURN_CODE 77) + + # Licence table of scripts/publish_hf.py (no network, no model needed). + add_test(NAME check_publish_hf_licences COMMAND ${PARAKEET_PY} ${CMAKE_SOURCE_DIR}/tests/python/check_publish_hf_licences.py) endif() pk_add_test(test_speaker_profiles) diff --git a/tests/python/check_publish_hf_licences.py b/tests/python/check_publish_hf_licences.py new file mode 100644 index 0000000..924d901 --- /dev/null +++ b/tests/python/check_publish_hf_licences.py @@ -0,0 +1,79 @@ +#!/usr/bin/env python3 +"""Check scripts/publish_hf.py licence handling. + +* every model id the script can publish has an explicit licence (no default); +* an unknown id raises instead of getting a default licence; +* the script's LICENSES table equals the table in docs/licenses.md; +* the generated collection card keeps the per-model licences, the required + notices and the sections for files that the script does not convert. + +No network, no conversion. Exit codes (ctest convention): 0 = pass, 1 = fail. +""" +import importlib.util +import re +import sys +from pathlib import Path + +root = Path(__file__).resolve().parent.parent.parent +spec = importlib.util.spec_from_file_location("publish_hf", root / "scripts" / "publish_hf.py") +ph = importlib.util.module_from_spec(spec) +spec.loader.exec_module(ph) + +failures = [] + + +def check(cond, msg): + if not cond: + failures.append(msg) + + +# 1. Explicit licence for every known id. +for problem in ph.check_licences(): + failures.append(problem) + +# 2. No silent default. +try: + ph._license("someone/unknown-model") + failures.append("unknown model id did not raise") +except KeyError: + pass + +# 3. Script table == docs table. +doc_rows = {} +for line in (root / "docs" / "licenses.md").read_text(encoding="utf-8").splitlines(): + m = re.match(r"\| `([^`]+)` \| ([^|]+?) \| <([^>]+)> \| ([^|]*?) \|$", line) + if m: + doc_rows[m.group(1)] = (m.group(2), m.group(3), m.group(4)) +script_rows = {k: (v[1], v[2], ph.LICENSE_NOTICES.get(k, "")) for k, v in ph.LICENSES.items()} +check(doc_rows == script_rows, "LICENSES differs from docs/licenses.md: " + f"{sorted(set(doc_rows.items()) ^ set(script_rows.items()))}") + +# 4. Generated card content. +card = ph.build_collection_card(ph.ALL_MODELS, ph.DEFAULT_VARIANTS, {}, ph.DEFAULT_COLLECTION_REPO) +front = card.split("---")[1] +check("license: other" in front, "front matter is not `license: other`") +check("license_name: mixed-per-model-see-license-section" in front, "license_name missing") +for needle in ( + "Licensed by NVIDIA Corporation under the NVIDIA Open Model License", + "Copyright (c) 2020-present Silero Team", + "https://openmdw.ai/license/1-1/", + "https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/", + "### nemotron-3-diarization", "### ultra and redux (Moondream)", "### silero-vad", + "### nemotron-3.5-asr-streaming-0.6b", + "snakers4/silero-vad", "moondream/parakeet-redux", "nvidia/Nemotron-3-Diarization", +): + check(needle in card, f"card is missing: {needle}") +section = re.search(r"^### realtime_eou_120m-v1\n\n(.*?)\n\n", card, re.S | re.M) +check(section is not None and "NVIDIA Open Model License" in section.group(1), + "realtime_eou_120m-v1 section lacks the NVIDIA Open Model License") +check("`realtime_eou_120m-v1-*`: [NVIDIA Open Model License]" in card, + "License section lacks the realtime_eou_120m-v1 bullet") +# CC-BY-4.0 must not be claimed for the non-CC models. +for bad in ("`realtime_eou_120m-v1-*`: [CC-BY-4.0]", "`nemotron-3.5-asr-streaming-0.6b-*`: [CC-BY-4.0]"): + check(bad not in card, f"wrong licence in card: {bad}") + +if failures: + for f in failures: + print(f"FAIL: {f}", file=sys.stderr) + sys.exit(1) +print("check_publish_hf_licences: PASS")