diff --git a/README.md b/README.md
index 3ad96b1..b1c0c27 100644
--- a/README.md
+++ b/README.md
@@ -559,4 +559,4 @@ Ettore Di Giacinto ([@mudler](https://github.com/mudler)).
## License
-parakeet.cpp is released under the [MIT License](LICENSE). The model weights are governed by the licenses of the original models, so check each model card on HuggingFace. The NVIDIA Parakeet models are mostly CC-BY-4.0 (nemotron-3.5-asr-streaming is OpenMDW-1.1). Moondream's [parakeet-ultra](https://huggingface.co/moondream/parakeet-ultra) and [parakeet-redux](https://huggingface.co/moondream/parakeet-redux) are CC-BY-4.0 too: credit Moondream and NVIDIA ([parakeet-tdt-0.6b-v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3)), link the [license](https://creativecommons.org/licenses/by/4.0/), and note that GGUF files made here are converted (and quantized, or dequantized for Redux) copies, not retrained models.
+parakeet.cpp is released under the [MIT License](LICENSE). The model weights are governed by the licenses of the original models, so check each model card on HuggingFace. The NVIDIA Parakeet models are mostly CC-BY-4.0 (nemotron-3.5-asr-streaming and Nemotron-3-Diarization are OpenMDW-1.1, parakeet_realtime_eou_120m-v1 is the NVIDIA Open Model License, and the Silero VAD files are MIT). See [docs/licenses.md](docs/licenses.md) for the full table. Moondream's [parakeet-ultra](https://huggingface.co/moondream/parakeet-ultra) and [parakeet-redux](https://huggingface.co/moondream/parakeet-redux) are CC-BY-4.0 too: credit Moondream and NVIDIA ([parakeet-tdt-0.6b-v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3)), link the [license](https://creativecommons.org/licenses/by/4.0/), and note that GGUF files made here are converted (and quantized, or dequantized for Redux) copies, not retrained models.
diff --git a/docs/licenses.md b/docs/licenses.md
new file mode 100644
index 0000000..b24dbfb
--- /dev/null
+++ b/docs/licenses.md
@@ -0,0 +1,38 @@
+# Model licences
+
+The GGUF files published at `mudler/parakeet-cpp-gguf` are converted from other
+people's models, so each file family keeps the licence of its source model. The
+parakeet.cpp code is MIT-licensed.
+
+This table is the reference for `scripts/publish_hf.py`. The script has no default
+licence: a model id that is missing from its `LICENSES` table is an error. The
+check `tests/python/check_publish_hf_licences.py` (and
+`python3 scripts/publish_hf.py --check-licences`) fails when this table and the
+script differ, so change both together.
+
+| Source checkpoint | Licence | Licence text | Notice to keep |
+|---|---|---|---|
+| `moondream/parakeet-redux` | CC-BY-4.0 | | |
+| `moondream/parakeet-ultra` | CC-BY-4.0 | | |
+| `nvidia/Nemotron-3-Diarization` | OpenMDW 1.1 | | |
+| `nvidia/nemotron-3.5-asr-streaming-0.6b` | OpenMDW 1.1 | | |
+| `nvidia/parakeet-ctc-0.6b` | CC-BY-4.0 | | |
+| `nvidia/parakeet-ctc-1.1b` | CC-BY-4.0 | | |
+| `nvidia/parakeet-rnnt-0.6b` | CC-BY-4.0 | | |
+| `nvidia/parakeet-rnnt-1.1b` | CC-BY-4.0 | | |
+| `nvidia/parakeet-tdt-0.6b-v2` | CC-BY-4.0 | | |
+| `nvidia/parakeet-tdt-0.6b-v3` | CC-BY-4.0 | | |
+| `nvidia/parakeet-tdt-1.1b` | CC-BY-4.0 | | |
+| `nvidia/parakeet-tdt_ctc-1.1b` | CC-BY-4.0 | | |
+| `nvidia/parakeet-tdt_ctc-110m` | CC-BY-4.0 | | |
+| `nvidia/parakeet_realtime_eou_120m-v1` | NVIDIA Open Model License | | Licensed by NVIDIA Corporation under the NVIDIA Open Model License |
+| `snakers4/silero-vad` | MIT | | Copyright (c) 2020-present Silero Team. Released under the MIT license |
+
+Notes:
+
+- The Moondream Ultra and Redux files are converted here, not trained. Credit
+ Moondream and NVIDIA (`parakeet-tdt-0.6b-v3`) when you use them. The Redux F16 and
+ Q8_0 files are dequantized from the ternary weights.
+- The Silero VAD files are converted from the official ONNX model, not trained.
+- Static card text for the files that `publish_hf.py` does not convert (diarization,
+ Moondream, Silero VAD, and the Nemotron 3.5 usage note) lives in `scripts/hf_card/`.
diff --git a/scripts/hf_card/diarization.md b/scripts/hf_card/diarization.md
new file mode 100644
index 0000000..ef30eaf
--- /dev/null
+++ b/scripts/hf_card/diarization.md
@@ -0,0 +1,14 @@
+### nemotron-3-diarization
+
+Source: [nvidia/Nemotron-3-Diarization](https://huggingface.co/nvidia/Nemotron-3-Diarization) · Speaker diarization (Sortformer, up to 8 speakers, streaming speaker cache) · License: [OpenMDW 1.1](https://openmdw.ai/license/1-1/)
+
+| File | Variant | Size | Segments vs NeMo |
+|---|---|---:|---:|
+| `nemotron-3-diarization-f16.gguf` ← **recommended** | F16 | 200.7 MB | identical |
+| `nemotron-3-diarization-q8_0.gguf` | Q8_0 | 108.7 MB | identical on 2 of 3 clips, 99.5% of frames on the third |
+
+> Checked against NeMo on a 23.6 s and a 68.5 s two-speaker clip (offline and streaming: every segment identical, to the 10 ms frame) and a 12.3 min three-speaker clip (F16 100%, Q8_0 99.9% of speech frames). Q8_0 can flip a frame whose probability sits right at the 0.5 threshold, splitting a segment (99.5% of frames on a 31.5 s clip); use F16 when segment-exact output matters. Answers "who spoke when"; pair it with any ASR model above for speaker-attributed transcripts. See [diarization.md](https://github.com/mudler/parakeet.cpp/blob/master/docs/diarization.md).
+
+```bash
+build/examples/cli/diarize models/nemotron-3-diarization-f16.gguf meeting.wav
+```
diff --git a/scripts/hf_card/moondream.md b/scripts/hf_card/moondream.md
new file mode 100644
index 0000000..8c697ca
--- /dev/null
+++ b/scripts/hf_card/moondream.md
@@ -0,0 +1,25 @@
+### ultra and redux (Moondream)
+
+Source: [moondream/parakeet-ultra](https://huggingface.co/moondream/parakeet-ultra) and [moondream/parakeet-redux](https://huggingface.co/moondream/parakeet-redux) by [Moondream](https://huggingface.co/moondream), derived from NVIDIA's [nvidia/parakeet-tdt-0.6b-v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) · TDT transducer (FastConformer), multilingual, with a voice-activity head (`transcribe --vad`) · heads: TDT · license: [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)
+
+| File | Variant | Size | Runs on |
+|---|---|---:|---|
+| `ultra-f16.gguf` | F16 | 1441.9 MB | any backend |
+| `ultra-q8_0.gguf` | Q8_0 | 941.5 MB | any backend |
+| `redux-packed.gguf` | packed ternary encoder | 213.3 MB | CPU only, offline only |
+| `redux-f16.gguf` | F16, dequantized | 1441.9 MB | any backend |
+| `redux-q8_0.gguf` | Q8_0, dequantized | 941.5 MB | any backend |
+
+> There is no NeMo reference for these models, so there is no WER column. Each file was checked to decode the `tests/fixtures/speech.wav` clip to the expected sentence. See [ternary.md](https://github.com/mudler/parakeet.cpp/blob/master/docs/ternary.md) for the measurements.
+
+- **Ultra** has ordinary F16 weights and runs on any backend, like the other v3 files.
+- **Redux packed** keeps the encoder linear layers as ternary weights (-1, 0 or +1 times a per-group scale) and runs a native CPU kernel. It does not run on GPU backends or in streaming mode, and parakeet.cpp refuses to load it there. For GPU use, take `redux-f16.gguf` or `redux-q8_0.gguf`.
+- **Redux F16 and Q8_0** are dequantized: the ternary weights were expanded to ordinary weights and then stored as F16 or Q8_0. They are not packed.
+- **Changes:** these files are converted here, not trained. Nothing was trained or fine-tuned by the parakeet.cpp project. The models were trained by NVIDIA (the base) and Moondream (Ultra and Redux).
+
+```bash
+huggingface-cli download mudler/parakeet-cpp-gguf redux-packed.gguf --local-dir models/
+build/examples/cli/parakeet-cli transcribe --model models/redux-packed.gguf --input audio.wav
+# Long audio: cut at pauses with the model's voice-activity head (offline only).
+build/examples/cli/parakeet-cli transcribe --model models/ultra-q8_0.gguf --input long.wav --vad
+```
diff --git a/scripts/hf_card/nemotron-3.5-note.md b/scripts/hf_card/nemotron-3.5-note.md
new file mode 100644
index 0000000..13eec2e
--- /dev/null
+++ b/scripts/hf_card/nemotron-3.5-note.md
@@ -0,0 +1,6 @@
+> This model runs both offline and as a cache-aware streaming model, and it is the only one here that takes a target language. Pass `--lang ` (for example `en-US`, `de-DE`, `es-ES`, `ja-JP`), or leave it at the default `auto` to let the model detect the language. The WER column is measured against NeMo, offline, on the languages en, de and auto; streaming and offline output also match NeMo for en, de, es, ja-JP and auto. See [parity.md](https://github.com/mudler/parakeet.cpp/blob/master/docs/parity.md). The small prompt layers, the LSTM and the feature extractor stay F32 in every quantization.
+
+```bash
+huggingface-cli download mudler/parakeet-cpp-gguf nemotron-3.5-asr-streaming-0.6b-f16.gguf --local-dir models/
+build/examples/cli/parakeet-cli transcribe --model models/nemotron-3.5-asr-streaming-0.6b-f16.gguf --input audio.wav --lang de-DE
+```
diff --git a/scripts/hf_card/silero-vad.md b/scripts/hf_card/silero-vad.md
new file mode 100644
index 0000000..cfa5462
--- /dev/null
+++ b/scripts/hf_card/silero-vad.md
@@ -0,0 +1,13 @@
+### silero-vad (Silero Team)
+
+Source: [snakers4/silero-vad](https://github.com/snakers4/silero-vad) v6.2.3 by the [Silero Team](https://github.com/snakers4) · a small voice-activity detector, not a speech recognizer · 8 kHz and 16 kHz in one file · license: [MIT](https://github.com/snakers4/silero-vad/blob/master/LICENSE)
+
+| File | Variant | Size |
+|---|---|---:|
+| `silero-vad-f32.gguf` | F32 | 2.2 MB |
+| `silero-vad-f16.gguf` | F16 weights, widened to F32 at load | 1.3 MB |
+
+- The files are converted from the official ONNX model with `scripts/convert_silero_vad_to_gguf.py` in parakeet.cpp. The GGUF records the source version and the ONNX sha256. They were converted here, not trained.
+- They need a parakeet.cpp build that has the standalone VAD API. That code is not in a release yet, so check the parakeet.cpp repository before relying on it.
+- Use it as a stand-alone detector (`parakeet-cli vad --model silero-vad-f16.gguf --input audio.wav`) or to cut long audio before transcribing with any model, including those that have no VAD head (`parakeet-cli transcribe --model tdt-0.6b-v3-q8_0.gguf --input long.wav --vad --vad-model silero-vad-f16.gguf`).
+- Probabilities match onnxruntime to about 1e-6 (F32) and 1e-3 (F16) on the test clips.
diff --git a/scripts/publish_hf.py b/scripts/publish_hf.py
index bc5eaf0..671c851 100644
--- a/scripts/publish_hf.py
+++ b/scripts/publish_hf.py
@@ -121,33 +121,134 @@
"q8_0": {"wer": 0.0, "size_mb": None},
"q4_k": {"wer": None, "size_mb": None},
},
- # Multilingual prompt-conditioned streaming model. WER measured offline
- # (en/de/auto) by scripts/e2e_nemo_compare.py vs NeMo; all five variants are
- # byte-for-byte (WER 0.0). prompt_kernel + LSTM + featurizer tensors stay F32.
+ # Multilingual prompt-conditioned streaming model. The WER is agreement with
+ # NeMo, not accuracy on a speech corpus. Recorded evidence (see docs/parity.md,
+ # scripts/e2e_nemo_compare.py, benchmarks/results/nemotron/bench.json):
+ # * F32: 20 rows (speech.wav + clip.wav x en/de/es/ja-JP/auto x offline/stream), WER 0.
+ # * Q8_0: speech.wav, en, offline, WER 0 (bench.json).
+ # * F16: stated in docs/parity.md, no run artifact is kept in the repo.
+ # * Q6_K, Q5_K, Q4_K: no measurement is recorded anywhere, so no WER is claimed.
+ # prompt_kernel + LSTM + featurizer tensors stay F32.
"nvidia/nemotron-3.5-asr-streaming-0.6b": {
"f16": {"wer": 0.0, "size_mb": 1484.3},
"q8_0": {"wer": 0.0, "size_mb": 983.7},
- "q6_k": {"wer": 0.0, "size_mb": 855.7},
- "q5_k": {"wer": 0.0, "size_mb": 784.8},
- "q4_k": {"wer": 0.0, "size_mb": 718.1},
+ "q6_k": {"wer": None, "size_mb": 855.7},
+ "q5_k": {"wer": None, "size_mb": 784.8},
+ "q4_k": {"wer": None, "size_mb": 718.1},
},
}
-# Per-model SPDX license id + human label, used in the generated card frontmatter
-# and the License section. Defaults to CC-BY-4.0 (the NVIDIA NeMo Parakeet
-# checkpoints); nemotron-3.5-asr-streaming is released under OpenMDW-1.1.
-DEFAULT_LICENSE = ("cc-by-4.0", "CC-BY-4.0",
- "https://creativecommons.org/licenses/by/4.0/")
+# Per-model licence: (frontmatter license id, human label, licence url).
+#
+# Every model id that the script can publish MUST have an explicit entry here.
+# There is no default: an unknown id raises, so a new model can never silently
+# ship under the wrong licence. `--check-licences` and
+# tests/python/check_publish_hf_licences.py enforce this, and also check that
+# this table equals the table in docs/licenses.md.
+CC_BY_4_0 = ("cc-by-4.0", "CC-BY-4.0", "https://creativecommons.org/licenses/by/4.0/")
+OPENMDW_1_1 = ("other", "OpenMDW 1.1", "https://openmdw.ai/license/1-1/")
+NVIDIA_OML = (
+ "other", "NVIDIA Open Model License",
+ "https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/",
+)
+MIT_SILERO = ("mit", "MIT", "https://github.com/snakers4/silero-vad/blob/master/LICENSE")
+
LICENSES: dict = {
- "nvidia/nemotron-3.5-asr-streaming-0.6b": (
- "other", "OpenMDW-1.1", "https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b",
- ),
+ "nvidia/parakeet-tdt_ctc-110m": CC_BY_4_0,
+ "nvidia/parakeet_realtime_eou_120m-v1": NVIDIA_OML,
+ "nvidia/nemotron-3.5-asr-streaming-0.6b": OPENMDW_1_1,
+ "nvidia/parakeet-ctc-0.6b": CC_BY_4_0,
+ "nvidia/parakeet-rnnt-0.6b": CC_BY_4_0,
+ "nvidia/parakeet-tdt-0.6b-v2": CC_BY_4_0,
+ "nvidia/parakeet-tdt-0.6b-v3": CC_BY_4_0,
+ "nvidia/parakeet-ctc-1.1b": CC_BY_4_0,
+ "nvidia/parakeet-rnnt-1.1b": CC_BY_4_0,
+ "nvidia/parakeet-tdt-1.1b": CC_BY_4_0,
+ "nvidia/parakeet-tdt_ctc-1.1b": CC_BY_4_0,
+ # Published by other tooling, but they have sections on the same card.
+ "nvidia/Nemotron-3-Diarization": OPENMDW_1_1,
+ "moondream/parakeet-ultra": CC_BY_4_0,
+ "moondream/parakeet-redux": CC_BY_4_0,
+ "snakers4/silero-vad": MIT_SILERO,
+}
+
+# Text the licence requires the redistributor to keep, shown on the card.
+LICENSE_NOTICES: dict = {
+ "nvidia/parakeet_realtime_eou_120m-v1":
+ "Licensed by NVIDIA Corporation under the NVIDIA Open Model License",
+ "snakers4/silero-vad":
+ "Copyright (c) 2020-present Silero Team. Released under the MIT license",
+}
+
+# Models and files on the card that this script does not convert. Their card
+# sections are static text in scripts/hf_card/, so a regenerated card keeps them.
+# Each entry is (model id or ids, section file).
+EXTERNAL_SECTIONS: list = [
+ (("nvidia/Nemotron-3-Diarization",), "diarization.md"),
+ (("moondream/parakeet-ultra", "moondream/parakeet-redux"), "moondream.md"),
+ (("snakers4/silero-vad",), "silero-vad.md"),
+]
+# Extra text that follows the table of a converted model (note + usage).
+MODEL_EXTRA_SECTIONS: dict = {
+ "nvidia/nemotron-3.5-asr-streaming-0.6b": "nemotron-3.5-note.md",
}
+CARD_SECTIONS_DIR = SCRIPTS_DIR / "hf_card"
def _license(model_id: str) -> tuple[str, str, str]:
- """(frontmatter license id, human label, upstream/license url) for *model_id*."""
- return LICENSES.get(model_id, DEFAULT_LICENSE)
+ """(frontmatter license id, human label, licence url) for *model_id*.
+
+ Raises KeyError for an id with no explicit entry: there is no default.
+ """
+ try:
+ return LICENSES[model_id]
+ except KeyError:
+ raise KeyError(
+ f"no explicit licence for {model_id!r}: add it to LICENSES in "
+ "scripts/publish_hf.py and to docs/licenses.md"
+ ) from None
+
+
+def _all_known_ids() -> List[str]:
+ ids = list(ALL_MODELS) + list(DERIVED_MODELS)
+ for group, _f in EXTERNAL_SECTIONS:
+ ids += list(group)
+ return sorted(set(ids))
+
+
+def check_licences() -> List[str]:
+ """Return a list of problems with the licence table (empty = fine)."""
+ problems: List[str] = []
+ for m in _all_known_ids():
+ if m not in LICENSES:
+ problems.append(f"{m}: no explicit licence entry")
+ for m in LICENSES:
+ if m not in _all_known_ids():
+ problems.append(f"{m}: licence entry for an id the script does not know")
+ for m in DERIVED_MODELS:
+ base = DERIVED_MODELS[m]["base"]
+ if m in LICENSES and base in LICENSES and LICENSES[m] != LICENSES[base]:
+ problems.append(f"{m}: licence differs from its base {base}")
+ for m in LICENSE_NOTICES:
+ if m not in LICENSES:
+ problems.append(f"{m}: notice for an id with no licence")
+ if "nvidia/parakeet_realtime_eou_120m-v1" in LICENSES and \
+ "Licensed by NVIDIA Corporation under the NVIDIA Open Model License" \
+ not in LICENSE_NOTICES.get("nvidia/parakeet_realtime_eou_120m-v1", ""):
+ problems.append("realtime_eou_120m-v1: missing the required NVIDIA notice")
+ files = [f for _g, f in EXTERNAL_SECTIONS] + list(MODEL_EXTRA_SECTIONS.values())
+ for f in files:
+ if not (CARD_SECTIONS_DIR / f).is_file():
+ problems.append(f"missing card section file scripts/hf_card/{f}")
+ return problems
+
+
+def _section_text(name: str) -> str:
+ return (CARD_SECTIONS_DIR / name).read_text(encoding="utf-8").rstrip("\n")
+
+
+def _file_glob(model_id: str) -> str:
+ return f"{_model_slug(model_id).lower()}-*"
# Moondream's derivatives of NVIDIA parakeet-tdt-0.6b-v3. They are trained by
@@ -542,6 +643,9 @@ def build_model_card(
f"released under the [{lic_label}]({lic_url}) license. "
"The parakeet.cpp runtime is MIT-licensed."
)
+ if model_id in LICENSE_NOTICES:
+ lines.append("")
+ lines.append(f"Notice: {LICENSE_NOTICES[model_id]}.")
lines.append("")
return "\n".join(lines)
@@ -649,44 +753,87 @@ def publish_model(
# Single-repo collection: one repo, all models × variants as flat GGUFs
# ---------------------------------------------------------------------------
+def _license_section_lines(models: List[str]) -> List[str]:
+ """The card's `## License` section: one bullet per file family, then credits."""
+ all_ids = list(models)
+ for group, _f in EXTERNAL_SECTIONS:
+ all_ids += [m for m in group if m not in all_ids]
+
+ # Group file families by licence, in first-seen order.
+ groups: dict = {}
+ for m in all_ids:
+ groups.setdefault(_license(m), []).append(m)
+
+ lines = [
+ "## License",
+ "",
+ "Licences differ by model, so the front matter says `license: other`. Each file "
+ "family follows the licence of the model it was converted from. The files are "
+ "converted, not trained here. The parakeet.cpp runtime itself is MIT-licensed.",
+ "",
+ ]
+ for (_id, label, url), ids in groups.items():
+ globs = ", ".join(f"`{_file_glob(m)}`" for m in ids)
+ lines.append(f"- {globs}: [{label}]({url}).")
+ lines.append("")
+
+ notices = [f"{LICENSE_NOTICES[m]}" for m in all_ids if m in LICENSE_NOTICES]
+ if notices:
+ lines.append("Notices that the licences require:")
+ lines.append("")
+ for m in all_ids:
+ if m in LICENSE_NOTICES:
+ lines.append(f"- `{_file_glob(m)}`: {LICENSE_NOTICES[m]}.")
+ lines.append("")
+
+ derived = [m for m in all_ids if m in DERIVED_MODELS]
+ if derived:
+ bases = sorted({DERIVED_MODELS[m]["base"] for m in derived})
+ lines.append(
+ "`ultra-*.gguf` and `redux-*.gguf` are converted from "
+ "[moondream/parakeet-ultra](https://huggingface.co/moondream/parakeet-ultra) and "
+ "[moondream/parakeet-redux](https://huggingface.co/moondream/parakeet-redux) by "
+ "Moondream, which are derived from NVIDIA's "
+ + ", ".join(f"[{b.split('/')[-1]}](https://huggingface.co/{b})" for b in bases)
+ + ". Credit Moondream and NVIDIA when you use these files. They were converted "
+ "here, not trained, and the Redux F16 and Q8_0 files are dequantized from the "
+ "ternary weights."
+ )
+ lines.append("")
+ return lines
+
+
def build_collection_card(
models: List[str],
variants: List[str],
paths_by_model: dict[str, dict[str, Path]],
repo_id: str,
) -> str:
- """One combined model card for the whole collection repo."""
+ """One combined model card for the whole collection repo.
+
+ *models* are the models this script converted. Files published by other
+ tooling (diarization, Moondream, Silero VAD) get their sections from
+ scripts/hf_card/ so a regeneration keeps them.
+ """
lines: List[str] = []
- # YAML frontmatter — base_model accepts a list. Licenses can differ per model
- # (most are CC-BY-4.0; nemotron is OpenMDW-1.1), so when the collection mixes
- # licenses we declare `other` here and spell each one out in the License
- # section / per-model rows below.
- licenses = {_license(m) for m in models}
- if len(licenses) == 1:
- only_id, only_label, only_url = next(iter(licenses))
- lines = ["---", f"license: {only_id}"]
- if only_id == "other":
- lines += [f"license_name: {only_label}", f"license_link: {only_url}"]
- lines += ["library_name: parakeet.cpp", "tags:"]
- else:
- lines = [
- "---", "license: other",
- "license_name: mixed (see per-model licenses below)",
- f"license_link: https://huggingface.co/{repo_id}",
- "library_name: parakeet.cpp", "tags:",
- ]
+ # YAML frontmatter. The card always mixes licences, so it declares `other`
+ # and spells each licence out in the License section.
+ lines = [
+ "---", "license: other",
+ "license_name: mixed-per-model-see-license-section",
+ f"license_link: https://huggingface.co/{repo_id}#license",
+ "library_name: parakeet.cpp", "tags:",
+ ]
lines += [f" - {t}" for t in (
"automatic-speech-recognition", "asr", "parakeet",
- "gguf", "ggml", "cpp-inference", "nemo",
+ "gguf", "ggml", "cpp-inference", "nemo", "speaker-diarization",
)]
lines.append("pipeline_tag: automatic-speech-recognition")
lines.append("base_model:")
bases = list(models)
- for m in models:
- b = DERIVED_MODELS.get(m, {}).get("base")
- if b and b not in bases:
- bases.append(b)
+ for group, _f in EXTERNAL_SECTIONS:
+ bases += [m for m in group if m not in bases]
lines += [f" - {m}" for m in bases]
lines += ["---", ""]
@@ -705,10 +852,28 @@ def build_collection_card(
)
lines.append("")
- # Per-model sections with a variant table each.
+ # Per-model sections with a variant table each. Models with their own extra
+ # text (nemotron) come after the shared WER note, as on the published card.
lines.append("## Models")
lines.append("")
- for model_id in models:
+
+ def _wer_note() -> None:
+ lines.append(
+ "> WER (word error rate) is computed against the upstream NeMo reference on "
+ "`tests/fixtures/speech.wav` (LibriSpeech `2086-149220-0033`, ~7.4 s, English). "
+ "0.0 = byte-for-byte identical transcript. "
+ "See [parity.md](https://github.com/mudler/parakeet.cpp/blob/master/docs/parity.md) "
+ "and [quantization.md](https://github.com/mudler/parakeet.cpp/blob/master/docs/quantization.md)."
+ )
+ lines.append("")
+
+ ordered = [m for m in models if m not in MODEL_EXTRA_SECTIONS] + \
+ [m for m in models if m in MODEL_EXTRA_SECTIONS]
+ note_written = False
+ for model_id in ordered:
+ if model_id in MODEL_EXTRA_SECTIONS and not note_written:
+ _wer_note()
+ note_written = True
slug = _model_slug(model_id)
arch_desc, heads = _arch_info(model_id)
wer_data = KNOWN_WER.get(model_id, {})
@@ -721,13 +886,8 @@ def build_collection_card(
f"{arch_desc} · heads: {heads} · license: [{_lic_label}]({_lic_url})"
)
lines.append("")
- if model_id in DERIVED_MODELS:
- d = DERIVED_MODELS[model_id]
- lines.append(
- f"By [Moondream](https://huggingface.co/moondream), derived from NVIDIA's "
- f"[{d['base']}](https://huggingface.co/{d['base']}). Converted, not trained here. "
- f"{d['conversion']}"
- )
+ if model_id in LICENSE_NOTICES:
+ lines.append(f"Notice: {LICENSE_NOTICES[model_id]}.")
lines.append("")
lines.append("| File | Variant | Size | WER vs NeMo |")
lines.append("|---|---|---:|---:|")
@@ -740,15 +900,15 @@ def build_collection_card(
rec = " ← **recommended**" if v == "f16" else ""
lines.append(f"| `{fname}`{rec} | {v.upper()} | {size} | {wer} |")
lines.append("")
+ if model_id in MODEL_EXTRA_SECTIONS:
+ lines.append(_section_text(MODEL_EXTRA_SECTIONS[model_id]))
+ lines.append("")
+ if not note_written:
+ _wer_note()
- lines.append(
- "> WER (word error rate) is computed against the upstream NeMo reference on "
- "`tests/fixtures/speech.wav` (LibriSpeech `2086-149220-0033`, ~7.4 s, English). "
- "0.0 = byte-for-byte identical transcript. "
- "See [parity.md](https://github.com/mudler/parakeet.cpp/blob/main/docs/parity.md) "
- "and [quantization.md](https://github.com/mudler/parakeet.cpp/blob/main/docs/quantization.md)."
- )
- lines.append("")
+ for _group, fname in EXTERNAL_SECTIONS:
+ lines.append(_section_text(fname))
+ lines.append("")
# Quantization notes (shared).
lines.append("## Quantization notes")
@@ -782,29 +942,7 @@ def build_collection_card(
lines.append("```")
lines.append("")
- # License. Each GGUF inherits the license of its source checkpoint; list them
- # explicitly because the collection can mix licenses (most CC-BY-4.0, nemotron
- # OpenMDW-1.1).
- lines.append("## License")
- lines.append("")
- lines.append(
- "Each GGUF is derived from its upstream checkpoint and inherits that checkpoint's "
- "license. The files are converted, not trained here. The parakeet.cpp runtime itself "
- "is MIT-licensed."
- )
- lines.append("")
- lines.append("| Source checkpoint | License |")
- lines.append("|---|---|")
- for model_id in models:
- _lic_id, _lic_label, _lic_url = _license(model_id)
- credit = ""
- if model_id in DERIVED_MODELS:
- b = DERIVED_MODELS[model_id]["base"]
- credit = f" (Moondream, derived from NVIDIA [{b}](https://huggingface.co/{b}))"
- lines.append(
- f"| [{model_id}](https://huggingface.co/{model_id}){credit} | [{_lic_label}]({_lic_url}) |"
- )
- lines.append("")
+ lines += _license_section_lines(models)
return "\n".join(lines)
@@ -1009,8 +1147,26 @@ def main() -> int:
"resumable — pair with a watchdog that restarts on stall."
),
)
+ parser.add_argument(
+ "--check-licences",
+ action="store_true",
+ help=(
+ "Check that every model id the script can publish has an explicit "
+ "licence entry, print the table and exit (0 = fine). No conversion, "
+ "no HuggingFace calls."
+ ),
+ )
args = parser.parse_args()
+ if args.check_licences:
+ problems = check_licences()
+ for m in _all_known_ids():
+ lic = LICENSES.get(m)
+ print(f"{m}: {lic[1] if lic else 'MISSING'}")
+ for p in problems:
+ print(f"error: {p}", file=sys.stderr)
+ return 1 if problems else 0
+
# Resolve the model list ('all' → canonical set), de-duplicated, order-stable.
raw_models = args.model or []
models: List[str] = []
@@ -1019,6 +1175,15 @@ def main() -> int:
if one not in models:
models.append(one)
+ unlicensed = [m for m in models if m not in LICENSES]
+ if unlicensed:
+ print(
+ "error: no explicit licence for: " + ", ".join(unlicensed) + ". Add each id to "
+ "LICENSES in scripts/publish_hf.py (and docs/licenses.md) before publishing.",
+ file=sys.stderr,
+ )
+ return 2
+
variants: List[str] = [v.strip() for v in args.variants.split(",") if v.strip()]
output_dir = Path(args.output_dir)
upload: bool = args.upload
diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt
index 0c7342b..d922d6c 100644
--- a/tests/CMakeLists.txt
+++ b/tests/CMakeLists.txt
@@ -221,6 +221,9 @@ if(PARAKEET_PY)
add_test(NAME check_hf_convert COMMAND ${PARAKEET_PY} ${CMAKE_SOURCE_DIR}/tests/python/check_hf_convert.py)
set_tests_properties(check_hf_convert PROPERTIES SKIP_RETURN_CODE 77)
+
+ # Licence table of scripts/publish_hf.py (no network, no model needed).
+ add_test(NAME check_publish_hf_licences COMMAND ${PARAKEET_PY} ${CMAKE_SOURCE_DIR}/tests/python/check_publish_hf_licences.py)
endif()
pk_add_test(test_speaker_profiles)
diff --git a/tests/python/check_publish_hf_licences.py b/tests/python/check_publish_hf_licences.py
new file mode 100644
index 0000000..924d901
--- /dev/null
+++ b/tests/python/check_publish_hf_licences.py
@@ -0,0 +1,79 @@
+#!/usr/bin/env python3
+"""Check scripts/publish_hf.py licence handling.
+
+* every model id the script can publish has an explicit licence (no default);
+* an unknown id raises instead of getting a default licence;
+* the script's LICENSES table equals the table in docs/licenses.md;
+* the generated collection card keeps the per-model licences, the required
+ notices and the sections for files that the script does not convert.
+
+No network, no conversion. Exit codes (ctest convention): 0 = pass, 1 = fail.
+"""
+import importlib.util
+import re
+import sys
+from pathlib import Path
+
+root = Path(__file__).resolve().parent.parent.parent
+spec = importlib.util.spec_from_file_location("publish_hf", root / "scripts" / "publish_hf.py")
+ph = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(ph)
+
+failures = []
+
+
+def check(cond, msg):
+ if not cond:
+ failures.append(msg)
+
+
+# 1. Explicit licence for every known id.
+for problem in ph.check_licences():
+ failures.append(problem)
+
+# 2. No silent default.
+try:
+ ph._license("someone/unknown-model")
+ failures.append("unknown model id did not raise")
+except KeyError:
+ pass
+
+# 3. Script table == docs table.
+doc_rows = {}
+for line in (root / "docs" / "licenses.md").read_text(encoding="utf-8").splitlines():
+ m = re.match(r"\| `([^`]+)` \| ([^|]+?) \| <([^>]+)> \| ([^|]*?) \|$", line)
+ if m:
+ doc_rows[m.group(1)] = (m.group(2), m.group(3), m.group(4))
+script_rows = {k: (v[1], v[2], ph.LICENSE_NOTICES.get(k, "")) for k, v in ph.LICENSES.items()}
+check(doc_rows == script_rows, "LICENSES differs from docs/licenses.md: "
+ f"{sorted(set(doc_rows.items()) ^ set(script_rows.items()))}")
+
+# 4. Generated card content.
+card = ph.build_collection_card(ph.ALL_MODELS, ph.DEFAULT_VARIANTS, {}, ph.DEFAULT_COLLECTION_REPO)
+front = card.split("---")[1]
+check("license: other" in front, "front matter is not `license: other`")
+check("license_name: mixed-per-model-see-license-section" in front, "license_name missing")
+for needle in (
+ "Licensed by NVIDIA Corporation under the NVIDIA Open Model License",
+ "Copyright (c) 2020-present Silero Team",
+ "https://openmdw.ai/license/1-1/",
+ "https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/",
+ "### nemotron-3-diarization", "### ultra and redux (Moondream)", "### silero-vad",
+ "### nemotron-3.5-asr-streaming-0.6b",
+ "snakers4/silero-vad", "moondream/parakeet-redux", "nvidia/Nemotron-3-Diarization",
+):
+ check(needle in card, f"card is missing: {needle}")
+section = re.search(r"^### realtime_eou_120m-v1\n\n(.*?)\n\n", card, re.S | re.M)
+check(section is not None and "NVIDIA Open Model License" in section.group(1),
+ "realtime_eou_120m-v1 section lacks the NVIDIA Open Model License")
+check("`realtime_eou_120m-v1-*`: [NVIDIA Open Model License]" in card,
+ "License section lacks the realtime_eou_120m-v1 bullet")
+# CC-BY-4.0 must not be claimed for the non-CC models.
+for bad in ("`realtime_eou_120m-v1-*`: [CC-BY-4.0]", "`nemotron-3.5-asr-streaming-0.6b-*`: [CC-BY-4.0]"):
+ check(bad not in card, f"wrong licence in card: {bad}")
+
+if failures:
+ for f in failures:
+ print(f"FAIL: {f}", file=sys.stderr)
+ sys.exit(1)
+print("check_publish_hf_licences: PASS")