From e49023e8a68c13dd656a0a8732aa26651f78ee59 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 3 Oct 2026 23:56:28 +0000 Subject: [PATCH 1/2] Add a VAD-only GGUF slice and load it for VAD calls scripts/slice_vad_gguf.py copies from an Ultra or Redux GGUF only what the VAD path reads: the mel filterbank and window, the subsampler, the vad_head tensors and the encoder, preprocessor and vad config keys. It does not requantize, sets parakeet.arch to "vad" and records the parent name, file, size, architecture and SHA-256. The marker is needed because the loader expects a vocabulary and decoder. ModelLoader accepts arch "vad" without a vocabulary. The new Model::load_vad_only serves the slice, and parakeet-cli vad and the C-API VAD calls use it through a separate context slot. Model::load and DiarizationModel::load refuse the slice with a log message, and every other C-API call reports that the context holds a VAD-only model. Existing files load as before. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] --- AGENTS.md | 2 + docs/vad.md | 27 ++++++++++ examples/cli/main.cpp | 5 +- scripts/slice_vad_gguf.py | 108 ++++++++++++++++++++++++++++++++++++++ src/diarization.cpp | 3 ++ src/model.cpp | 27 ++++++++++ src/model.hpp | 7 +++ src/model_loader.cpp | 3 +- src/parakeet_capi.cpp | 54 ++++++++++++------- src/vad_json.cpp | 11 ++++ src/vad_json.hpp | 4 ++ tests/CMakeLists.txt | 2 + tests/test_vad_only.cpp | 67 +++++++++++++++++++++++ 13 files changed, 299 insertions(+), 21 deletions(-) create mode 100644 scripts/slice_vad_gguf.py create mode 100644 tests/test_vad_only.cpp diff --git a/AGENTS.md b/AGENTS.md index 7d7f8fc..1c67920 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -113,6 +113,7 @@ examples/cli/ parakeet-cli binary scripts/ Python tooling convert_parakeet_to_gguf.py, .nemo/.hf -> GGUF (--dtype f32|f16|q8_0) convert_hf_parakeet_to_gguf.py, HF safetensors (moondream/parakeet-ultra, -redux) to GGUF (--template, --ternary keep|dequant, --vad keep|drop) + slice_vad_gguf.py , cut a VAD-only GGUF (parakeet.arch "vad") out of an Ultra or Redux GGUF gen_nemo_baseline.py , NeMo intermediates -> baseline.gguf gen_stream_baseline.py , NeMo cache-aware streaming encode+decode -> stream baseline.gguf gen_diar_baseline.py , NeMo offline + streaming diarization -> diar baseline.gguf @@ -150,6 +151,7 @@ tests/ ctest targets test_vad_head.cpp , VAD head probabilities (PARAKEET_TEST_GGUF_ULTRA) test_vad_segmenter.cpp , segmenter cut rules, 32 ms grid, Silero defaults, event tracker (model-independent) test_vad_options.cpp , VAD option parser, NULL and bad-file C-API paths (model-independent) + test_vad_only.cpp , VAD-only slice: C-API VAD equals the parent, other calls refuse it (PARAKEET_TEST_VAD_ONLY_GGUF, PARAKEET_TEST_GGUF_ULTRA) test_capi_vad_silero.cpp, Silero via the C-API: JSON, options, threads, stream (PARAKEET_TEST_SILERO_GGUF) test_transcribe_vad_silero.cpp, transcribe with Silero segments on a model without a head (PARAKEET_TEST_SILERO_GGUF, PARAKEET_TEST_GGUF) test_transcribe_vad.cpp , --vad path vs plain pass on long audio (PARAKEET_TEST_GGUF_ULTRA, PARAKEET_TEST_GGUF) diff --git a/docs/vad.md b/docs/vad.md index 2c33f08..466396c 100644 --- a/docs/vad.md +++ b/docs/vad.md @@ -136,6 +136,33 @@ mode (frame period 0.032 s) and the same decode of each segment as the head path. Audio of at most 30 s is transcribed whole and the VAD does not run. Passing NULL as the second argument uses the model's own head. +## VAD-only slice + +The VAD head of Ultra and Redux reads only the log-mel front end, the subsampler +and the head, which is about 10 MB of the 213 MB to 1.4 GB file. `scripts/slice_vad_gguf.py` +copies exactly those tensors, byte for byte, into a small GGUF: + + python scripts/slice_vad_gguf.py ultra-q8_0.gguf ultra-vad.gguf + +Nothing is requantized, so the VAD output is the same as the parent's. The slice +copies the `parakeet.encoder.*`, `parakeet.preprocessor.*` and `parakeet.vad.*` +keys. It sets `parakeet.arch` to `vad` (`general.architecture` stays `parakeet`), +and it records the parent in `parakeet.vad_only.parent_name`, `parent_file`, +`parent_sha256`, `parent_bytes` and `parent_arch`. + +The `vad` marker is what the loader needs: a normal file has an encoder, +a decoder and a vocabulary, and the slice has none of them. `parakeet-cli vad` +and the `parakeet_capi_vad_*` functions accept the file (`Model::load_vad_only`). +`Model::load` refuses it with a log message, so every other call (transcribe, +diarize, stream) fails with "context holds a VAD-only model" or a load error. +Files that are not slices load exactly as before. + +The two Redux parents (packed ternary and dequantized F16) give slices with the same +tensors: the subsampler and the head are not ternary. The Ultra Q8_0 parent has a +Q8_0 final subsampler projection, so its slice is 6.0 MB; the F16 parents give +9.9 MB. A slice from a Q8_0 parent and a slice from an F16 parent give different +probabilities because their weights differ, not because of the slicing. + ## ABI All of this is additive. `parakeet_capi_abi_version` stays 10. A new symbol set diff --git a/examples/cli/main.cpp b/examples/cli/main.cpp index 16e1945..36c6d1e 100644 --- a/examples/cli/main.cpp +++ b/examples/cli/main.cpp @@ -1901,7 +1901,7 @@ static int cmd_vad_probe(int argc, char** argv) { // [--min-pause SEC] [--min-speech SEC] [--speech-pad SEC] [--max-segment SEC=30] // [--mode speech|segments] [--probabilities] [--threads N] // Prints the same JSON as parakeet_capi_vad_path_json. The model is an ASR GGUF -// with a VAD head (Ultra, Redux) or a Silero VAD GGUF. Unset options keep the +// with a VAD head (Ultra, Redux), a VAD-only slice of one, or a Silero VAD GGUF. Unset options keep the // defaults of that model kind. static int cmd_vad(int argc, char** argv) { std::string model, input; @@ -1974,7 +1974,8 @@ static int cmd_vad(int argc, char** argv) { std::printf("%s\n", pk::silero_vad_to_json(*sv, audio.samples, 16000, req).c_str()); return 0; } - std::unique_ptr m = pk::Model::load(model); + std::unique_ptr m = + pk::gguf_is_vad_only(model) ? pk::Model::load_vad_only(model) : pk::Model::load(model); if (!m) { std::fprintf(stderr, "parakeet-cli: failed to load model %s\n", model.c_str()); return 1; } std::printf("%s\n", pk::vad_to_json(*m, audio.samples, req).c_str()); } catch (const std::exception& e) { diff --git a/scripts/slice_vad_gguf.py b/scripts/slice_vad_gguf.py new file mode 100644 index 0000000..0889bc7 --- /dev/null +++ b/scripts/slice_vad_gguf.py @@ -0,0 +1,108 @@ +#!/usr/bin/env python3 +"""Cut a VAD-only GGUF out of an Ultra or Redux parakeet GGUF. + +moondream/parakeet-ultra and -redux carry a voice-activity head on the +encoder's subsampler output. The VAD path reads only the log-mel filterbank and +window, the subsampler (`encoder.pre_encode.*`) and the head (`vad_head.*`). +This script copies exactly those tensors, byte for byte, plus the config keys +that path reads. Nothing is requantized, so the slice produces the same VAD +output as its parent. + +The output sets `parakeet.arch` to `vad`. The loader needs that marker: a +normal file has an encoder, a decoder and a vocabulary, and the slice has none +of them. With the marker, `parakeet-cli vad` and the C-API VAD calls load the +file, and every other entry point refuses it with a clear message. +`general.architecture` stays `parakeet`. + +Provenance keys written (all strings or integers): + parakeet.vad_only.parent_name general.name of the parent + parakeet.vad_only.parent_file file name of the parent + parakeet.vad_only.parent_sha256 SHA-256 of the parent file + parakeet.vad_only.parent_bytes size of the parent file + parakeet.vad_only.parent_arch parakeet.arch of the parent (for example tdt) + + python scripts/slice_vad_gguf.py ultra-q8_0.gguf ultra-vad.gguf +""" +import argparse +import hashlib +import pathlib +import sys + +try: + import gguf +except ImportError as e: # pragma: no cover - env guard + print(f"slice_vad_gguf: missing dependency: {e}", file=sys.stderr) + sys.exit(2) + +TENSOR_PREFIXES = ("vad_head.", "encoder.pre_encode.", "preprocessor.featurizer.") +KEY_PREFIXES = ("parakeet.encoder.", "parakeet.preprocessor.", "parakeet.vad.") +# Packed ternary tensors cannot appear on the VAD path; refuse them loudly. +PACKED_SUFFIXES = (".weight_packed", ".weight_scales") + + +def sha256_file(path): + h = hashlib.sha256() + with open(path, "rb") as f: + for blk in iter(lambda: f.read(1 << 22), b""): + h.update(blk) + return h.hexdigest() + + +def field_value(f): + return f.contents() + + +def main(): + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("src", help="Ultra or Redux GGUF with a VAD head") + ap.add_argument("dst", help="output VAD-only GGUF") + a = ap.parse_args() + + r = gguf.GGUFReader(a.src) + flds = r.fields + present = flds.get("parakeet.vad.present") + if present is None or not bool(field_value(present)): + sys.exit("slice_vad_gguf: source has no VAD head (parakeet.vad.present is not true)") + arch = flds.get("parakeet.arch") + if arch is not None and str(field_value(arch)) == "vad": + sys.exit("slice_vad_gguf: source is already a VAD-only file") + + tensors = [t for t in r.tensors if t.name.startswith(TENSOR_PREFIXES)] + names = {t.name for t in tensors} + for need in ("vad_head.proj.weight", "vad_head.proj.bias", "vad_head.ctx.weight", "vad_head.ctx.bias", + "vad_head.out.weight", "vad_head.out.bias", "preprocessor.featurizer.fb", + "preprocessor.featurizer.window", "encoder.pre_encode.out.weight"): + if need not in names: + sys.exit(f"slice_vad_gguf: source lacks tensor {need}") + for t in tensors: + if t.name.endswith(PACKED_SUFFIXES): + sys.exit(f"slice_vad_gguf: packed tensor {t.name} on the VAD path; not supported") + + parent_name = str(field_value(flds["general.name"])) if "general.name" in flds else pathlib.Path(a.src).name + w = gguf.GGUFWriter(a.dst, "parakeet") + w.add_name(parent_name + " (VAD only)") + w.add_string("parakeet.arch", "vad") + for k, f in flds.items(): + if not k.startswith(KEY_PREFIXES): + continue + # Copy with the original GGUF value type so the C++ readers see the same kind. + vt = f.types[0] + sub = f.types[-1] if vt == gguf.GGUFValueType.ARRAY else None + w.add_key_value(k, field_value(f), vt, sub_type=sub) + w.add_string("parakeet.vad_only.parent_name", parent_name) + w.add_string("parakeet.vad_only.parent_file", pathlib.Path(a.src).name) + w.add_string("parakeet.vad_only.parent_sha256", sha256_file(a.src)) + w.add_uint64("parakeet.vad_only.parent_bytes", pathlib.Path(a.src).stat().st_size) + w.add_string("parakeet.vad_only.parent_arch", str(field_value(arch)) if arch is not None else "") + for t in tensors: + w.add_tensor(t.name, t.data, raw_shape=t.data.shape, raw_dtype=t.tensor_type) + w.write_header_to_file() + w.write_kv_data_to_file() + w.write_tensors_to_file() + w.close() + n = sum(t.n_bytes for t in tensors) + print(f"wrote {a.dst}: {len(tensors)} tensors, {n} tensor bytes, from {a.src}") + + +if __name__ == "__main__": + main() diff --git a/src/diarization.cpp b/src/diarization.cpp index 64f9493..3358522 100644 --- a/src/diarization.cpp +++ b/src/diarization.cpp @@ -3,6 +3,7 @@ #include "audio_io.hpp" #include "backend.hpp" +#include "common.hpp" #include "ggml_graph.hpp" #include @@ -21,6 +22,8 @@ std::unique_ptr DiarizationModel::load(const std::string& path const auto& cfg = m->loader_.config(); if (cfg.arch != "diarization") { // Not a diarization model — caller should use Model for ASR. + if (cfg.arch == "vad") + PK_LOG("%s is a VAD-only file and cannot diarize", path.c_str()); return nullptr; } if (!cfg.diarization.present) { diff --git a/src/model.cpp b/src/model.cpp index 322e951..3df57fd 100644 --- a/src/model.cpp +++ b/src/model.cpp @@ -52,6 +52,11 @@ std::unique_ptr Model::load(const std::string& gguf_path) { if (!m->loader_.load(gguf_path)) { return nullptr; } + if (m->loader_.config().arch == "vad") { + PK_LOG("%s is a VAD-only file: it has no encoder or decoder, so it cannot transcribe or diarize; " + "use it with `parakeet-cli vad` or the parakeet_capi_vad_* functions", gguf_path.c_str()); + return nullptr; + } // Model is the ASR entry point — reject diarization models so the C-API // can fall through to DiarizationModel::load. if (m->loader_.config().arch == "diarization") { @@ -90,6 +95,28 @@ std::unique_ptr Model::load(const std::string& gguf_path) { return m; } +std::unique_ptr Model::load_vad_only(const std::string& gguf_path) { + std::unique_ptr m(new (std::nothrow) Model()); + if (!m) return nullptr; + if (!m->loader_.load(gguf_path)) return nullptr; + const ParakeetConfig& c = m->loader_.config(); + if (c.arch != "vad") { + PK_LOG("%s is not a VAD-only file (parakeet.arch is \"%s\")", gguf_path.c_str(), c.arch.c_str()); + return nullptr; + } + if (!c.vad.present) { + PK_LOG("%s: VAD-only file without parakeet.vad.present", gguf_path.c_str()); + return nullptr; + } + // The slice never carries packed ternary tensors (the slicer refuses them). + if (c.ternary.present || m->loader_.has_tensor_with_suffix(".weight_packed")) { + PK_LOG("%s: a VAD-only file cannot hold packed ternary tensors", gguf_path.c_str()); + return nullptr; + } + ensure_weights_realized(m->loader_); + return m; +} + std::shared_ptr Model::pool_snapshot() const { std::lock_guard lk(pool_mu_); return pool_; diff --git a/src/model.hpp b/src/model.hpp index 345695a..9d0e84c 100644 --- a/src/model.hpp +++ b/src/model.hpp @@ -42,8 +42,15 @@ struct NBestTranscription { class Model { public: // Loads the GGUF at `gguf_path`. Returns nullptr on failure (no throw). + // A VAD-only slice (parakeet.arch "vad") is refused here with a log message: + // it holds no encoder blocks or decoder. Use load_vad_only(). static std::unique_ptr load(const std::string& gguf_path); + // Loads a VAD-only slice made by scripts/slice_vad_gguf.py. Returns nullptr + // on failure or when the file is not a slice. The result serves + // vad_probabilities() and config() only; every other call is invalid. + static std::unique_ptr load_vad_only(const std::string& gguf_path); + // Transcribe raw mono float PCM. If `sample_rate != 16000` the audio is // linearly resampled to 16 kHz (via pk::resample_linear) before inference. // `target_lang` selects the language prompt for multilingual (nemotron) diff --git a/src/model_loader.cpp b/src/model_loader.cpp index 54861d7..0ee1cee 100644 --- a/src/model_loader.cpp +++ b/src/model_loader.cpp @@ -278,7 +278,8 @@ bool ModelLoader::load(const std::string& path){ const int64_t nt = gguf_get_n_tensors(gguf_); for(int64_t i=0;i0 && (cfg_.vocab_size>0 || cfg_.arch=="diarization"); + // A VAD-only slice (arch "vad", see scripts/slice_vad_gguf.py) has no vocabulary. + return cfg_.d_model>0 && (cfg_.vocab_size>0 || cfg_.arch=="diarization" || cfg_.arch=="vad"); } ggml_tensor* ModelLoader::tensor(const std::string& n) const { auto it = tensors_.find(n); return it==tensors_.end()? nullptr : it->second; diff --git a/src/parakeet_capi.cpp b/src/parakeet_capi.cpp index da8c2e0..cff3ba9 100644 --- a/src/parakeet_capi.cpp +++ b/src/parakeet_capi.cpp @@ -95,12 +95,13 @@ class ErrorSlot { }; // The opaque context: a loaded model plus a buffer for the last error message. -// Exactly one of `model` / `diar` / `tagger` / `speaker` / `silero` is non-null: +// Exactly one of `model` / `vad_model` / `diar` / `tagger` / `speaker` / `silero` is non-null: // ASR models use `model`, diarization models (Sortformer) use `diar`, CED // sound-event taggers use `tagger`, voice-detect speaker encoders use `speaker`, // Silero VAD models use `silero`. struct parakeet_ctx { std::unique_ptr model; + std::unique_ptr vad_model; // VAD-only slice (parakeet.arch "vad"): VAD calls only std::unique_ptr diar; std::unique_ptr tagger; std::unique_ptr speaker; @@ -237,6 +238,14 @@ char* dup_to_c(const std::string& s) { } // namespace +namespace { +// Why a call that needs an ASR or diarization model found none. +const char* no_model_msg(const parakeet_ctx* c) { + return c->vad_model ? "context holds a VAD-only model: only parakeet_capi_vad_* calls work with it" + : "context has no loaded model"; +} +} // namespace + extern "C" int parakeet_capi_abi_version(void) { return PARAKEET_CAPI_ABI_VERSION; } @@ -265,6 +274,15 @@ extern "C" parakeet_ctx* parakeet_capi_load(const char* gguf_path) { return nullptr; } + // A VAD-only slice of an Ultra or Redux model (parakeet.arch "vad"): only + // the VAD calls accept it. + if (pk::gguf_is_vad_only(gguf_path)) { + ctx->vad_model = pk::Model::load_vad_only(gguf_path); + if (ctx->vad_model) return ctx; + delete ctx; + return nullptr; + } + // A voice-detect GGUF (architecture "voicedetect") is a speaker encoder. if (pk::gguf_is_voicedetect(gguf_path)) { try { @@ -312,7 +330,7 @@ extern "C" char* parakeet_capi_transcribe_path_lang(parakeet_ctx* ctx, if (!ctx->model) { ctx->last_error = ctx->diar ? "context holds a diarization model; use parakeet_capi_diarize_*" - : "context has no loaded model"; + : no_model_msg(ctx); return nullptr; } if (!wav_path) { ctx->last_error = "wav_path is NULL"; return nullptr; } @@ -347,7 +365,7 @@ extern "C" char* parakeet_capi_transcribe_pcm_lang(parakeet_ctx* ctx, if (!ctx->model) { ctx->last_error = ctx->diar ? "context holds a diarization model; use parakeet_capi_diarize_*" - : "context has no loaded model"; + : no_model_msg(ctx); return nullptr; } if (!samples || n_samples < 0) { ctx->last_error = "invalid samples buffer"; return nullptr; } @@ -389,7 +407,7 @@ extern "C" int parakeet_capi_transcribe_pcm_logits(parakeet_ctx* ctx, *out_logits = nullptr; *out_T = 0; *out_vocab_plus_1 = 0; - if (!ctx->model) { ctx->last_error = "context has no loaded model"; return 1; } + if (!ctx->model) { ctx->last_error = no_model_msg(ctx); return 1; } if (!samples || n_samples < 0) { ctx->last_error = "invalid samples buffer"; return 1; } try { std::vector pcm(samples, samples + n_samples); @@ -429,7 +447,7 @@ extern "C" int parakeet_capi_transcribe_pcm_batch_lang(parakeet_ctx* ctx, if (!ctx->model) { ctx->last_error = ctx->diar ? "context holds a diarization model; use parakeet_capi_diarize_*" - : "context has no loaded model"; + : no_model_msg(ctx); return 1; } if (!samples || !n_samples || !out || n_clips < 0) { @@ -491,7 +509,7 @@ extern "C" char* parakeet_capi_transcribe_path_json(parakeet_ctx* ctx, if (!ctx->model) { ctx->last_error = ctx->diar ? "context holds a diarization model; use parakeet_capi_diarize_*" - : "context has no loaded model"; + : no_model_msg(ctx); return nullptr; } if (!wav_path) { ctx->last_error = "wav_path is NULL"; return nullptr; } @@ -523,7 +541,7 @@ extern "C" char* parakeet_capi_transcribe_path_json_vad(parakeet_ctx* ctx, if (!ctx->model) { ctx->last_error = ctx->diar ? "context holds a diarization model; use parakeet_capi_diarize_*" - : "context has no loaded model"; + : no_model_msg(ctx); return nullptr; } if (!wav_path) { ctx->last_error = "wav_path is NULL"; return nullptr; } @@ -561,14 +579,14 @@ char* vad_json_common(parakeet_ctx* ctx, const std::vector& pcm, int rate const pk::VadKind kind = ctx->silero ? pk::VadKind::kSilero : pk::VadKind::kHead; if (!pk::parse_vad_options(options_json, req, err, kind)) { ctx->last_error = err; return nullptr; } std::string json = ctx->silero ? pk::silero_vad_to_json(*ctx->silero, pcm, rate, req) - : pk::vad_to_json(*ctx->model, pcm, req); + : pk::vad_to_json(ctx->model ? *ctx->model : *ctx->vad_model, pcm, req); ctx->last_error.clear(); char* out = dup_to_c(json); if (!out) { ctx->last_error = "out of memory"; return nullptr; } return out; } bool vad_ctx_ok(parakeet_ctx* ctx) { - if (ctx->model || ctx->silero) return true; + if (ctx->model || ctx->vad_model || ctx->silero) return true; ctx->last_error = ctx->diar ? "context holds a diarization model; use parakeet_capi_diarize_*" : "context has no loaded model or VAD model"; @@ -629,7 +647,7 @@ extern "C" char* parakeet_capi_transcribe_path_json_vad_with(parakeet_ctx* ctx, if (!ctx->model) { ctx->last_error = ctx->diar ? "context holds a diarization model; use parakeet_capi_diarize_*" - : "context has no loaded model"; + : no_model_msg(ctx); return nullptr; } if (!wav_path) { ctx->last_error = "wav_path is NULL"; return nullptr; } @@ -791,7 +809,7 @@ extern "C" char* parakeet_capi_transcribe_pcm_batch_json_lang(parakeet_ctx* ctx, if (!ctx->model) { ctx->last_error = ctx->diar ? "context holds a diarization model; use parakeet_capi_diarize_*" - : "context has no loaded model"; + : no_model_msg(ctx); return nullptr; } if (!samples_concat || !n_samples || n_clips < 0) { @@ -844,7 +862,7 @@ extern "C" char* parakeet_capi_transcribe_path_nbest_json( int beam_size, int nbest, int score_norm, const char* target_lang) { if (!ctx) return nullptr; if (!ctx->model) { - ctx->last_error = "context has no loaded model"; + ctx->last_error = no_model_msg(ctx); return nullptr; } if (!wav_path) { @@ -884,7 +902,7 @@ extern "C" char* parakeet_capi_transcribe_pcm_nbest_json( int beam_size, int nbest, int score_norm, const char* target_lang) { if (!ctx) return nullptr; if (!ctx->model) { - ctx->last_error = "context has no loaded model"; + ctx->last_error = no_model_msg(ctx); return nullptr; } if (!samples || n_samples < 0) { @@ -1006,7 +1024,7 @@ extern "C" parakeet_stream* parakeet_capi_stream_begin_lang(parakeet_ctx* ctx, if (!ctx->model) { ctx->last_error = ctx->diar ? "context holds a diarization model; use parakeet_capi_diarize_*" - : "context has no loaded model"; + : no_model_msg(ctx); return nullptr; } if (!ctx->model->config().streaming.present) { @@ -1299,7 +1317,7 @@ bool require_diar(parakeet_ctx* ctx) { ctx->last_error = ctx->model ? "context holds an ASR model; diarize_* needs a diarization model" : ctx->tagger ? "context holds a CED sound model; diarize_* needs a diarization model" : ctx->speaker ? "context holds a speaker model; diarize_* needs a diarization model" - : "context has no loaded model"; + : no_model_msg(ctx); return false; } return true; @@ -1311,7 +1329,7 @@ bool require_asr(parakeet_ctx* ctx) { ctx->last_error = ctx->diar ? "context holds a diarization model; an ASR model is needed here" : ctx->tagger ? "context holds a CED sound model; an ASR model is needed here" : ctx->speaker ? "context holds a speaker model; an ASR model is needed here" - : "context has no loaded model"; + : no_model_msg(ctx); return false; } return true; @@ -1326,7 +1344,7 @@ bool require_tagger(parakeet_ctx* ctx) { ctx->last_error = ctx->model ? "context holds an ASR model; a CED sound model is needed here" : ctx->diar ? "context holds a diarization model; a CED sound model is needed here" : ctx->speaker ? "context holds a speaker model; a CED sound model is needed here" - : "context has no loaded model"; + : no_model_msg(ctx); return false; } return true; @@ -1341,7 +1359,7 @@ bool require_speaker(parakeet_ctx* ctx) { ctx->last_error = ctx->model ? "context holds an ASR model; a speaker model is needed here" : ctx->diar ? "context holds a diarization model; a speaker model is needed here" : ctx->tagger ? "context holds a CED sound model; a speaker model is needed here" - : "context has no loaded model"; + : no_model_msg(ctx); return false; } return true; diff --git a/src/vad_json.cpp b/src/vad_json.cpp index 3c7b17b..9d3ce06 100644 --- a/src/vad_json.cpp +++ b/src/vad_json.cpp @@ -189,4 +189,15 @@ bool gguf_is_silero(const std::string& path) { return ok; } +bool gguf_is_vad_only(const std::string& path) { + gguf_init_params ip{/*no_alloc=*/true, /*ctx=*/nullptr}; + gguf_context* g = gguf_init_from_file(path.c_str(), ip); + if (!g) return false; + const int64_t id = gguf_find_key(g, "parakeet.arch"); + const bool ok = id >= 0 && gguf_get_kv_type(g, id) == GGUF_TYPE_STRING && + std::string(gguf_get_val_str(g, id)) == "vad"; + gguf_free(g); + return ok; +} + } // namespace pk diff --git a/src/vad_json.hpp b/src/vad_json.hpp index 0c4100f..db38a0d 100644 --- a/src/vad_json.hpp +++ b/src/vad_json.hpp @@ -38,6 +38,10 @@ std::string silero_vad_to_json(const SileroVad& m, const std::vector& pcm // True when the GGUF's general.architecture is "silero_vad". Reads only the header. bool gguf_is_silero(const std::string& gguf_path); +// True when the GGUF's parakeet.arch is "vad": a VAD-only slice of an Ultra or +// Redux model (scripts/slice_vad_gguf.py). Reads only the header. +bool gguf_is_vad_only(const std::string& gguf_path); + // Runs the VAD head over 16 kHz mono PCM and returns the JSON document // {"mode","duration","frame_sec","backend","segments":[{"start","end"}], // "probabilities":[...]?}. Throws std::runtime_error("model has no VAD head") diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index d922d6c..e84b45e 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -236,6 +236,8 @@ pk_add_test(test_silero_vad) pk_add_test(test_vad_options) pk_add_test(test_capi_vad_silero) pk_add_test(test_transcribe_vad_silero) +pk_add_test(test_vad_only) target_compile_definitions(test_silero_vad PRIVATE PK_SOURCE_DIR="${CMAKE_SOURCE_DIR}") set_tests_properties(test_silero_vad test_capi_vad_silero test_transcribe_vad_silero PROPERTIES LABELS "model") set_tests_properties(test_capi_vad_silero test_transcribe_vad_silero PROPERTIES WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}) +set_tests_properties(test_vad_only PROPERTIES LABELS "model") diff --git a/tests/test_vad_only.cpp b/tests/test_vad_only.cpp new file mode 100644 index 0000000..ce513c7 --- /dev/null +++ b/tests/test_vad_only.cpp @@ -0,0 +1,67 @@ +// A VAD-only slice (parakeet.arch "vad", made by scripts/slice_vad_gguf.py): +// the C-API VAD calls give the same JSON as the parent model, and every other +// entry point refuses the file with a clear message. Plain ASR loading is +// unchanged: Model::load refuses the slice. +// +// LABEL model; run from the project root. +// Env: PARAKEET_TEST_VAD_ONLY_GGUF slice of the parent below (skip 77 if unset) +// PARAKEET_TEST_GGUF_ULTRA the parent Ultra/Redux GGUF (skip 77 if unset) +#include "parakeet_capi.h" +#include "audio_io.hpp" +#include "model.hpp" + +#include +#include +#include +#include +#include +#include + +static int failures = 0; +#define CHECK(c) do { if (!(c)) { std::fprintf(stderr, "FAIL: %s (line %d)\n", #c, __LINE__); ++failures; } } while (0) + +static std::string take(char* p) { + if (!p) return std::string(); + std::string s(p); + parakeet_capi_free_string(p); + return s; +} + +int main() { + const char* slice = std::getenv("PARAKEET_TEST_VAD_ONLY_GGUF"); + const char* parent = std::getenv("PARAKEET_TEST_GGUF_ULTRA"); + if (!slice || !parent) { std::printf("SKIP: PARAKEET_TEST_VAD_ONLY_GGUF / PARAKEET_TEST_GGUF_ULTRA not set\n"); return 77; } + + // Three seconds of a tone burst between silences: enough to run the path. + std::vector pcm(16000 * 4, 0.0f); + for (size_t i = 16000; i < 16000 * 3; ++i) pcm[i] = 0.3f * std::sin(0.05f * (float)i) * std::sin(0.0007f * (float)i); + + parakeet_ctx* s = parakeet_capi_load(slice); + parakeet_ctx* f = parakeet_capi_load(parent); + CHECK(s && f); + if (!s || !f) return 1; + + const char* opts = "{\"probabilities\":true}"; + const std::string js = take(parakeet_capi_vad_pcm_json(s, pcm.data(), (int)pcm.size(), 16000, opts)); + const std::string jf = take(parakeet_capi_vad_pcm_json(f, pcm.data(), (int)pcm.size(), 16000, opts)); + CHECK(!js.empty()); + CHECK(js == jf); + + // Other entry points refuse the slice and name the reason. + CHECK(parakeet_capi_transcribe_path(s, "x.wav", 0) == nullptr); + const char* e = parakeet_capi_last_error(s); + CHECK(e && std::strstr(e, "VAD-only")); + CHECK(parakeet_capi_diarize_path(s, "x.wav") == nullptr); + e = parakeet_capi_last_error(s); + CHECK(e && std::strstr(e, "VAD-only")); + + // The ASR loader refuses the slice, and the VAD-only loader refuses a full model. + CHECK(pk::Model::load(slice) == nullptr); + CHECK(pk::Model::load_vad_only(parent) == nullptr); + + parakeet_capi_free(s); + parakeet_capi_free(f); + if (failures) { std::fprintf(stderr, "%d failure(s)\n", failures); return 1; } + std::printf("OK\n"); + return 0; +} From b2417fa5cf85dd2feb85d3f2ea34cd309da16b9e Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 4 Oct 2026 01:36:18 +0000 Subject: [PATCH 2/2] docs: link the published VAD-only slices Add the two published slice files, redux-vad.gguf and ultra-vad-q8_0.gguf, with sizes and SHA-256 to docs/vad.md, the README and models/MANIFEST.md. Note that they are cut out of Moondream's models (CC-BY-4.0) and cannot transcribe. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] --- README.md | 6 ++++++ docs/vad.md | 15 +++++++++++++++ models/MANIFEST.md | 7 +++++++ 3 files changed, 28 insertions(+) diff --git a/README.md b/README.md index b1c0c27..05c1218 100644 --- a/README.md +++ b/README.md @@ -84,6 +84,7 @@ ternary ones. - Silero VAD (MIT, 32 ms frames, 16 kHz and 8 kHz) runs from its own small GGUF ([download](https://huggingface.co/mudler/parakeet-cpp-gguf)) through the same functions, as a stream (`parakeet_capi_vad_stream_*`), and as the cutter for any ASR model: `parakeet-cli transcribe --vad --vad-model silero.gguf`. See [`docs/vad.md`](docs/vad.md). +- VAD-only slices of Ultra and Redux (6 to 10 MB, `redux-vad.gguf` and `ultra-vad-q8_0.gguf` in the same repo) hold just the head and its front end. They run `vad` and the `parakeet_capi_vad_*` calls and cannot transcribe. See [`docs/vad.md`](docs/vad.md). --- ## Performance @@ -295,6 +296,11 @@ parakeet-cli transcribe --model ultra.gguf --input long.wav --vad # Tune with --threshold F (0.5), --min-pause SEC (0.2), --min-speech SEC (0.1), --max-segment SEC (30) parakeet-cli vad --model ultra.gguf --input audio.wav +# Only the VAD head is needed? Use a 6 to 10 MB slice instead of the full model +# (it cannot transcribe). Files and checksums: docs/vad.md. +curl -LO https://huggingface.co/mudler/parakeet-cpp-gguf/resolve/main/redux-vad.gguf +parakeet-cli vad --model redux-vad.gguf --input audio.wav + # The same with a Silero VAD GGUF (frame_sec 0.032; defaults 250 ms min speech, # 100 ms min pause, 30 ms pad). Any ASR model can then cut long audio with it. # Download it from the collection repo (F16 is 1.3 MB, F32 is 2.2 MB). To make the diff --git a/docs/vad.md b/docs/vad.md index 466396c..0155674 100644 --- a/docs/vad.md +++ b/docs/vad.md @@ -157,6 +157,21 @@ and the `parakeet_capi_vad_*` functions accept the file (`Model::load_vad_only`) diarize, stream) fails with "context holds a VAD-only model" or a load error. Files that are not slices load exactly as before. +### Published slices + +Two slices are published in [`mudler/parakeet-cpp-gguf`](https://huggingface.co/mudler/parakeet-cpp-gguf). +They are cut out of Moondream's models, not trained here, and carry the same +CC-BY-4.0 license: credit Moondream and NVIDIA. + +| File | Parent | Size (bytes) | SHA-256 | +|---|---|---:|---| +| [`redux-vad.gguf`](https://huggingface.co/mudler/parakeet-cpp-gguf/resolve/main/redux-vad.gguf) | parakeet-redux | 9,939,488 | `588e1d6e2ee5b6cdfd9ec5ea98dc0993d5bea498d9cc4ec8d6077041eef8a34f` | +| [`ultra-vad-q8_0.gguf`](https://huggingface.co/mudler/parakeet-cpp-gguf/resolve/main/ultra-vad-q8_0.gguf) | parakeet-ultra, Q8_0 | 6,007,328 | `8b891a4435e97438104ca07c72530d0c5fe62b986baee48b2dd4e1500c1d4758` | + + parakeet-cli vad --model redux-vad.gguf --input audio.wav + +A slice cannot transcribe, diarize or stream. + The two Redux parents (packed ternary and dequantized F16) give slices with the same tensors: the subsampler and the head are not ternary. The Ultra Q8_0 parent has a Q8_0 final subsampler projection, so its slice is 6.0 MB; the F16 parents give diff --git a/models/MANIFEST.md b/models/MANIFEST.md index 38fe9f1..cd1f962 100644 --- a/models/MANIFEST.md +++ b/models/MANIFEST.md @@ -129,6 +129,13 @@ or AVX-512 VNNI and aarch64 with dotprod get SIMD kernels for the packed file. T original packed tensors resident next to the repacked planes, so its memory use is more than 213 MB. Ultra Q8_0 is measured in `docs/ternary.md`. +VAD-only slices (made with `scripts/slice_vad_gguf.py`, cut from the Ultra and Redux files above, not retrained; CC-BY-4.0, credit Moondream and NVIDIA). They run VAD only and cannot transcribe: + +| File | Parent | Size (bytes) | SHA-256 | +|---|---|---:|---| +| [`redux-vad.gguf`](https://huggingface.co/mudler/parakeet-cpp-gguf/resolve/main/redux-vad.gguf) | redux | 9,939,488 | `588e1d6e2ee5b6cdfd9ec5ea98dc0993d5bea498d9cc4ec8d6077041eef8a34f` | +| [`ultra-vad-q8_0.gguf`](https://huggingface.co/mudler/parakeet-cpp-gguf/resolve/main/ultra-vad-q8_0.gguf) | ultra, `--dtype q8_0` | 6,007,328 | `8b891a4435e97438104ca07c72530d0c5fe62b986baee48b2dd4e1500c1d4758` | + Both carry the `parakeet.vad.*` KVs and `vad_head.*` tensors unless converted with `--vad drop`.