docs: VAD benchmarks (Silero vs the Parakeet head vs whisper.cpp) with reproducible scripts

[localai-org-maint-bot]

What

Adds docs/vad-benchmarks.md, one page with all the VAD measurements made so far, and scripts/vad_bench/ with the scripts, small result files and load logs behind them. The page is linked from docs/vad.md and the README. No audio and no model files are committed.

It covers:

• The Parakeet VAD head (Ultra Q8_0, Redux packed) against Silero: 216 synthetic LibriSpeech clips in 9 noise conditions, and three TED-LIUM talks. Frame precision, recall and F1, boundary error, agreement, a threshold sweep.
• Silero in whisper.cpp against Silero in parakeet.cpp and onnxruntime: probability parity, segment quality, agreement, speed and memory.
• The batched decode of VAD segments on four long talks (the measurement behind perf: decode VAD segments in batches #82).
• The Silero parity table of feat: standalone VAD (model head and Silero), streaming, and external VAD for transcribe #83.
• The stand-alone head ("slice" GGUF of feat: VAD-only GGUF files (head, front end, subsampler) and a slicer #87): file size, load time, peak memory and speed against the full ASR file and Silero F16. feat: VAD-only GGUF files (head, front end, subsampler) and a slicer #87 is open, so the slice code is not in a release.
• A "when to use which" section limited to what the numbers support.

Each section gives how the data was built, the metric definitions, the tables, the caveats and the commands to repeat it.

Where the numbers come from

Every number was copied from saved result files of the runs, or computed from the saved per-run timings. The result files that are small enough are in scripts/vad_bench/results/. The raw per-clip predictions are not committed.

Not quiet-machine numbers

All timings were taken on a shared machine that was never idle. The page says this next to each timing: the Silero speed runs started at a 1-minute load average of 4.2 to 15.9, the batched decode runs at 2.5 to 18.0, and the slice runs at up to 17. The 8-thread numbers of the slice run are noisy (rounds of one configuration differ by up to 30%). A re-run on a quiet machine was tried: the load average was checked 19 times over 3 hours and was never below 5, so nothing was re-run. The poll log is in the results directory. Accuracy numbers do not depend on load.

Limits and what could not be traced

The page marks these as unverified:

• The script that cut the LibriSpeech subset of the heads-against-Silero run was not saved, so that corpus can be rebuilt in kind but not byte for byte. The commit of that run's parakeet build was not recorded.
• Which three TED talks were used, and why one more talk was left out.
• Whether the 393x Silero ONNX speed is a best or a median value, and thread pinning for that run.
• The build commits of the three builds in the batched decode runs (taken from the description of perf: decode VAD segments in batches #82).
• The 300 s clip of the Silero speed runs was not kept; it is assumed to be the first 300 s of the 600 s talk.

References are coarse: the synthetic reference counts pauses inside utterances as speech, and the TED reference is built from ASR word times of a Parakeet model, so a bias toward the Parakeet heads is possible. There are no confidence intervals. Differences below the noise are not claimed. Peak memory for a 600 s clip, GPU and streaming were not measured.

Checks

Docs and scripts only. The Python scripts compile, the shell script passes bash -n, and the wvad.cpp test program compiles against whisper.cpp. The data and comparison scripts were not run end to end after being cleaned of local paths.

🤖 Generated with Claude Code

[localai-org-maint-bot]

Update: #87 is merged (e53a253), so the sentence in the description saying #87 is open no longer applies. This branch was rebased onto master. The slice code is in master but not in a tagged release yet (latest is v0.5.0), and the benchmarks page now links to the slice section of docs/vad.md instead of repeating it.