From 33457da0486b2a8be76eed0f129c5397cefaa592 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 3 Oct 2026 22:46:55 +0000 Subject: [PATCH 1/4] docs: publish the VAD benchmarks Add docs/vad-benchmarks.md with every VAD measurement made so far: the Parakeet heads against Silero on synthetic and TED-LIUM audio, Silero in whisper.cpp against Silero in parakeet.cpp, and the batched decode of VAD segments on long talks. Each section gives the data, the metrics, the numbers, the caveats and how to repeat the run. The scripts, small result files and load logs are in scripts/vad_bench. Link the page from docs/vad.md and the README. The stand-alone memory of the head and the timing re-run on a quiet machine are marked pending. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] --- README.md | 2 + docs/vad-benchmarks.md | 495 ++++++++++++++++++ docs/vad.md | 5 +- scripts/vad_bench/README.md | 69 +++ scripts/vad_bench/analyze_synthetic.py | 62 +++ scripts/vad_bench/compare_synthetic.py | 50 ++ scripts/vad_bench/compare_ted.py | 51 ++ scripts/vad_bench/fetch_data.py | 47 ++ scripts/vad_bench/fetch_ted_talks.py | 18 + scripts/vad_bench/heads_lib.py | 60 +++ scripts/vad_bench/longform_b1.sh | 41 ++ scripts/vad_bench/make_clips.py | 31 ++ scripts/vad_bench/onnx_time.py | 10 + scripts/vad_bench/pk.py | 26 + .../results/heads_synthetic_signed.txt | 11 + .../results/heads_synthetic_tables.txt | 33 ++ .../results/heads_ted_and_speed_output.txt | 8 + .../vad_bench/results/longform_b1_loadlog.txt | 127 +++++ .../vad_bench/results/longform_b1_results.tsv | 120 +++++ .../results/silero_parity_results.txt | 50 ++ .../vad_bench/results/silero_speed2_raw.json | 110 ++++ .../vad_bench/results/silero_speed_raw.json | 242 +++++++++ .../results/silero_versions_and_sha256.txt | 16 + scripts/vad_bench/silero_analyze.py | 93 ++++ scripts/vad_bench/silero_collect.py | 41 ++ scripts/vad_bench/speed_heads.py | 18 + scripts/vad_bench/speed_silero.py | 71 +++ scripts/vad_bench/ted_reference.py | 19 + scripts/vad_bench/vad_synth.py | 46 ++ scripts/vad_bench/wvad.cpp | 65 +++ 30 files changed, 2036 insertions(+), 1 deletion(-) create mode 100644 docs/vad-benchmarks.md create mode 100644 scripts/vad_bench/README.md create mode 100755 scripts/vad_bench/analyze_synthetic.py create mode 100755 scripts/vad_bench/compare_synthetic.py create mode 100755 scripts/vad_bench/compare_ted.py create mode 100755 scripts/vad_bench/fetch_data.py create mode 100755 scripts/vad_bench/fetch_ted_talks.py create mode 100755 scripts/vad_bench/heads_lib.py create mode 100755 scripts/vad_bench/longform_b1.sh create mode 100755 scripts/vad_bench/make_clips.py create mode 100755 scripts/vad_bench/onnx_time.py create mode 100755 scripts/vad_bench/pk.py create mode 100644 scripts/vad_bench/results/heads_synthetic_signed.txt create mode 100644 scripts/vad_bench/results/heads_synthetic_tables.txt create mode 100644 scripts/vad_bench/results/heads_ted_and_speed_output.txt create mode 100644 scripts/vad_bench/results/longform_b1_loadlog.txt create mode 100644 scripts/vad_bench/results/longform_b1_results.tsv create mode 100644 scripts/vad_bench/results/silero_parity_results.txt create mode 100644 scripts/vad_bench/results/silero_speed2_raw.json create mode 100644 scripts/vad_bench/results/silero_speed_raw.json create mode 100644 scripts/vad_bench/results/silero_versions_and_sha256.txt create mode 100755 scripts/vad_bench/silero_analyze.py create mode 100755 scripts/vad_bench/silero_collect.py create mode 100755 scripts/vad_bench/speed_heads.py create mode 100755 scripts/vad_bench/speed_silero.py create mode 100755 scripts/vad_bench/ted_reference.py create mode 100755 scripts/vad_bench/vad_synth.py create mode 100644 scripts/vad_bench/wvad.cpp diff --git a/README.md b/README.md index 05c1218..5621ca0 100644 --- a/README.md +++ b/README.md @@ -85,6 +85,8 @@ ternary ones. functions, as a stream (`parakeet_capi_vad_stream_*`), and as the cutter for any ASR model: `parakeet-cli transcribe --vad --vad-model silero.gguf`. See [`docs/vad.md`](docs/vad.md). - VAD-only slices of Ultra and Redux (6 to 10 MB, `redux-vad.gguf` and `ultra-vad-q8_0.gguf` in the same repo) hold just the head and its front end. They run `vad` and the `parakeet_capi_vad_*` calls and cannot transcribe. See [`docs/vad.md`](docs/vad.md). +- Accuracy, speed and size of both detectors, with the method and the scripts to repeat + them: [`docs/vad-benchmarks.md`](docs/vad-benchmarks.md). --- ## Performance diff --git a/docs/vad-benchmarks.md b/docs/vad-benchmarks.md new file mode 100644 index 0000000..0ac1acb --- /dev/null +++ b/docs/vad-benchmarks.md @@ -0,0 +1,495 @@ +# VAD benchmarks + +This page lists every voice activity detection (VAD) measurement made for +parakeet.cpp so far: how each one was built, what it measures, the numbers, and +how to run it again. For the API and the model files, see [vad.md](vad.md). + +Read the caveats before you quote a number. Every reference here is coarse, the +machine was shared, and some runs were not on a quiet machine. A difference that +is smaller than the noise is not claimed. + +The detectors: + +| Name | What it is | +| --- | --- | +| Silero | Silero VAD 6.2.3 (MIT), 32 ms frames, from its own small GGUF (F32 or F16) | +| Ultra Q8_0 | VAD head of `moondream/parakeet-ultra`, GGUF in Q8_0, 80 ms frames | +| Redux packed | VAD head of `moondream/parakeet-redux`, packed ternary GGUF, 80 ms frames | +| whisper.cpp Silero | Silero VAD inside whisper.cpp (`whisper_vad_*`), used as a comparison | +| ONNX Silero | The official Silero ONNX file run with onnxruntime (CPU, one thread), used as the reference | + +## Summary + +Accuracy is the frame level F1 (percent) against a coarse reference, see +[Metrics](#metrics). Speed is audio seconds per wall second ("x real time") on a +300 s clip. Higher is faster. + +| Detector | F1, clean synthetic | F1, pink noise 0 dB | F1, TED-LIUM talks | Speed | Peak memory | File size | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| Silero (matched settings) | 94.1 | 91.7 | 91.8 | about 660x on 1 thread, `parakeet-cli vad`, includes process start and model load | 60 MB (`parakeet-cli vad`, F32 and F16) | 2.2 MB (F32), 1.3 MB (F16) | +| Silero (its own defaults) | 94.4 | 91.9 | 92.1 | not timed separately | same as above | same as above | +| Ultra Q8_0 head | 95.0 | 93.4 | 93.8 | 216x, 8 threads | pending, see below | 941.5 MB (the whole ASR model) | +| Redux packed head | 95.4 | 94.3 | 94.2 | 221x, 8 threads | pending, see below | 213.3 MB (the whole ASR model) | +| whisper.cpp Silero | 94.1 | not run | not run | in process: 686x on 1 thread | 36 MB (test harness) | 0.9 MB | +| ONNX Silero (onnxruntime) | 94.0 | not run | not run | in process: 449x on 1 thread, Python loop | 104 MB | 2.3 MB | + +How to read it: + +- The synthetic and TED columns come from the + [heads against Silero](#parakeet-heads-against-silero) benchmark. The whisper.cpp + and ONNX rows come from the [Silero parity](#silero-in-whispercpp-and-in-parakeetcpp) + benchmark, which uses a different synthetic corpus (120 clips, 3 conditions). The + clean F1 of "Silero" and "whisper.cpp Silero" are therefore not from the same clips. + Compare rows only inside one benchmark. +- "Matched settings" means Silero ran with the same segmentation settings as the + heads (threshold 0.5, minimum speech 100 ms, minimum silence 200 ms, no padding). + See [Metrics](#metrics). +- The head speed is the speed of a whole encoder pass on 8 threads, because the head + reads the encoder output. The four head builds that were timed (Ultra Q8_0 216x, + Ultra F16 218x, Redux packed 221x, Redux dequantized F16 222x) are within 3% of + each other. This is a different job from the one Silero does, so the speed columns + of Silero and the heads are not a like for like comparison of the models. They show + the cost of using each one as a stand alone gate. +- Memory of the head as a stand alone VAD is **pending**: it needs a measurement of + the head without the rest of the ASR model, see + [Slice only head](#slice-only-head-pending). +- File size of the heads is the size of the whole ASR GGUF the head lives in. A + stand alone head file is part of the pending measurement. + +### Machine and build + +| Item | Value | +| --- | --- | +| CPU | AMD Ryzen 9 class (Zen 5), 16-core desktop part; the OS shows 20 logical CPUs, one thread per core | +| Memory | 84 GB | +| OS and compiler | Linux 6.8, GCC 13.3, CMake 3.28 | +| ggml | commit e705c5fe (the submodule of this repository), `GGML_NATIVE=ON`, Release | +| parakeet.cpp | the VAD code of PR 83 (merged as `6165e3d`); see the per benchmark build notes below | +| whisper.cpp | commit `60c0be6ac8fa71b1a2ae2dd938a31a34a508e774` (version 1.9.4), CPU only | +| Load | the machine was shared with other jobs; load is given per benchmark | + +Build commits per benchmark: + +| Benchmark | parakeet.cpp build | +| --- | --- | +| Heads against Silero | a development build of the VAD branch; the commit was not recorded (unverified) | +| Silero in whisper.cpp and in parakeet.cpp | `c1a8391102d34074f48e758c441758327d2465b7`, the VAD branch before its squash into `6165e3d` | +| Long talks, batched decode | three builds, called base, b0 and b1: master before PR 81, PR 81 (`cbabaf5`), PR 82 (`9fdf3a1`). The result files do not record the commits; the mapping is from the PR 82 description (unverified) | +| Timing re-run | `6165e3d`, see [Timing re-run](#timing-re-run) | + +## Metrics + +All VAD output is reduced to speech regions (start and end in seconds). + +- **Frame level precision, recall, F1.** The regions of the detector and of the + reference are turned into a speech mask on a 10 ms grid. Counts of true positive, + false positive, false negative and true negative frames are summed over all clips + of a group, then precision, recall and F1 are computed from the sums. Pooled, not + an average of per clip scores. +- **Boundary error.** For each reference start (and end), the distance in ms to the + nearest detected start (end). Reported as median and 90th percentile, in ms, plus + the miss rate: the share of reference boundaries with no detected boundary within + 1 s. "All conditions" pools start and end errors. +- **Agreement.** Two detectors compared with each other: F1 of the two speech masks, + Cohen's kappa, and IoU (intersection over union) of the masks. +- **Probability parity.** Per frame, the absolute difference of the speech + probability against onnxruntime on the official ONNX file. Also the share of + frames where the decision at 0.5 flips. +- **Matched settings.** The segmentation step is not the same code in every system, + and the defaults differ. For a fair model comparison, Silero in the first + benchmark runs the `silero-vad` package (`get_speech_timestamps`) with threshold + 0.5, `min_speech_duration_ms` 100, `min_silence_duration_ms` 200, + `speech_pad_ms` 0. These are the head defaults. Silero's reference code also + lowers the threshold by 0.15 once speech has started (hysteresis); the parakeet.cpp + segmenter has no hysteresis (see [vad.md](vad.md)). "Silero (its own defaults)" uses + the package defaults (minimum speech 250 ms, minimum silence 100 ms, padding 30 ms). + The heads ran with their own defaults, which are the matched values. + +## Parakeet heads against Silero + +### Data + +Synthetic clips. Built from LibriSpeech test-clean (public). Every clip has five +utterances in a random order. Each utterance is trimmed at the first and last 10 ms +frame whose RMS is above 1% of the peak frame RMS. The utterances are joined with +gaps of 0.3 to 2.0 s (uniform), with 0.5 to 1.5 s before the first and after the +last. The gaps hold Gaussian noise at a standard deviation of 0.001. The reference +speech region of an utterance is the whole trimmed utterance, so short pauses inside +an utterance count as speech. + +Noise is added to the whole clip at a signal to noise ratio (SNR) measured against +the power of the speech regions: white or pink, at 20, 10, 5 and 0 dB, plus a clean +condition. That gives 9 conditions of 24 clips: **216 clips**, **1080 reference +regions**. The speech layout of clip k is the same in every condition. + +How the 24 clips were chosen is partly unverified. The script that cut the +LibriSpeech subset for this run was not saved. `scripts/vad_bench/fetch_data.py` +takes every 13th utterance of test-clean, which is the scheme of the Silero parity +corpus, and the numbers below were measured on a subset of that kind. A rebuild with +the script will give the same kind of corpus, not the same bytes, and the numbers +will differ slightly. + +TED-LIUM talks. Three talks from `distil-whisper/tedlium-long-form` (test split), +3178.6 s in total. Which three is not recorded in the result files (unverified). The +reference is built from word timestamps of the 0.6B v3 F16 ASR model run on 30 s +pieces: words closer than 0.4 s are merged into one region. This reference is +derived from an ASR model and is coarser than the synthetic one. It is also not +independent of the Parakeet family. The reference speech fraction is 91.9%. One more +talk was left out of the run by a name filter; the reason was not recorded. + +### Results: synthetic, threshold 0.5 + +Cells are precision / recall / F1, in percent. "Silero M" is Silero with matched +settings, "Silero D" with its own defaults. + +| Condition | Silero M | Silero D | Ultra Q8_0 | Redux packed | +| --- | --- | --- | --- | --- | +| clean | 99.1 / 89.7 / 94.1 | 98.7 / 90.5 / 94.4 | 98.2 / 92.0 / 95.0 | 98.4 / 92.7 / 95.4 | +| white 20 dB | 99.4 / 88.4 / 93.5 | 99.0 / 89.2 / 93.9 | 98.8 / 89.9 / 94.1 | 98.7 / 91.7 / 95.1 | +| white 10 dB | 99.3 / 88.0 / 93.3 | 98.9 / 89.0 / 93.7 | 98.9 / 88.3 / 93.3 | 98.2 / 91.9 / 94.9 | +| white 5 dB | 99.2 / 87.8 / 93.1 | 98.8 / 88.5 / 93.4 | 98.9 / 87.2 / 92.7 | 97.4 / 91.9 / 94.6 | +| white 0 dB | 99.2 / 85.8 / 92.0 | 98.8 / 86.7 / 92.4 | 98.3 / 87.9 / 92.8 | 96.4 / 92.3 / 94.3 | +| pink 20 dB | 99.4 / 88.4 / 93.6 | 99.1 / 89.3 / 93.9 | 98.8 / 90.8 / 94.6 | 98.6 / 92.0 / 95.2 | +| pink 10 dB | 99.3 / 88.0 / 93.3 | 99.0 / 89.0 / 93.7 | 99.0 / 88.8 / 93.6 | 98.1 / 92.2 / 95.1 | +| pink 5 dB | 99.4 / 87.6 / 93.1 | 99.0 / 88.5 / 93.5 | 99.0 / 88.4 / 93.4 | 97.4 / 92.6 / 94.9 | +| pink 0 dB | 99.2 / 85.2 / 91.7 | 98.8 / 85.9 / 91.9 | 96.3 / 90.8 / 93.4 | 95.5 / 93.2 / 94.3 | + +What the table supports: the three detectors are within about 1.5 F1 points of each +other at 5 dB and above, and up to 2.6 points apart at 0 dB. Silero has the higher precision and the lower recall. Both +heads have the higher recall. In the loudest noise (0 dB) the heads keep their F1 +better than Silero (pink 0 dB: Silero M 91.7, Ultra 93.4, Redux 94.3). The recall +of all detectors is below 94% because the reference counts pauses inside utterances +as speech, so the absolute recall values are not a measure of missed speech. + +Boundary error, in ms (median / 90th percentile, miss rate), start then end: + +| Condition | Silero M | Silero D | Ultra Q8_0 | Redux packed | +| --- | --- | --- | --- | --- | +| clean | S 171/527 (0.0%) E 121/366 (0.0%) | S 162/499 (0.0%) E 137/343 (0.0%) | S 123/460 (0.0%) E 168/325 (0.0%) | S 152/459 (0.0%) E 151/328 (0.0%) | +| white 10 dB | S 249/526 (0.0%) E 116/371 (0.0%) | S 232/500 (0.0%) E 129/357 (0.0%) | S 211/470 (0.0%) E 147/472 (0.8%) | S 163/466 (0.8%) E 152/389 (0.8%) | +| white 0 dB | S 262/571 (0.8%) E 117/459 (2.5%) | S 237/526 (0.8%) E 142/451 (1.7%) | S 162/461 (0.0%) E 149/494 (2.5%) | S 202/460 (0.8%) E 149/404 (1.7%) | +| pink 10 dB | S 248/526 (0.0%) E 106/384 (0.0%) | S 232/502 (0.0%) E 117/343 (0.0%) | S 226/479 (0.0%) E 136/445 (0.0%) | S 176/445 (0.0%) E 148/340 (0.0%) | +| pink 0 dB | S 261/570 (0.0%) E 145/411 (3.3%) | S 248/530 (0.0%) E 150/408 (3.3%) | S 180/457 (2.5%) E 162/470 (2.5%) | S 200/457 (1.7%) E 160/428 (1.7%) | + +All conditions pooled: + +| System | Median | 90th percentile | Start median | End median | Miss rate | +| --- | ---: | ---: | ---: | ---: | ---: | +| Silero M | 143 | 501 | 237 | 116 | 0.4% | +| Silero D | 149 | 484 | 232 | 130 | 0.3% | +| Ultra Q8_0 | 154 | 465 | 181 | 149 | 0.6% | +| Redux packed | 153 | 437 | 170 | 149 | 0.4% | + +Boundary errors of 100 to 250 ms are large next to the 32 and 80 ms frame sizes. +Most of this comes from the reference (trimmed utterances next to short gaps), so +only the ordering of start errors is informative: the heads place starts about 50 +to 70 ms closer to the reference than Silero M does. The signed start error median +is 237 ms for Silero M, 232 for Silero D, 162 for Ultra and 147 for Redux (the +detector starts later than the reference). The signed end error median is 3, 20, -10 +and 54 ms. Segments per reference region: Silero M 1.96, Ultra 1.92, Redux 1.76 +(a value above 1 means the detector splits utterances at their inner pauses). + +Threshold sweep, pooled over clean, white 10 dB, white 0 dB and pink 10 dB +(precision / recall / F1): + +| System | threshold 0.3 | threshold 0.5 | threshold 0.7 | +| --- | --- | --- | --- | +| Silero M | 98.5 / 90.3 / 94.3 | 99.2 / 87.9 / 93.2 | 99.5 / 86.2 / 92.4 | +| Ultra Q8_0 | 97.1 / 92.4 / 94.7 | 98.6 / 89.3 / 93.7 | 99.0 / 86.2 / 92.1 | +| Redux packed | 94.8 / 95.0 / 94.9 | 97.7 / 92.3 / 94.9 | 98.9 / 90.0 / 94.2 | + +Agreement with Silero M on all synthetic clips: Ultra F1 96.6, kappa 0.882, IoU 93.4. +Redux F1 96.3, kappa 0.865, IoU 92.8. Ultra against Redux: F1 97.2, kappa 0.896. + +### Results: TED-LIUM talks + +Frame level on 3178.6 s, with the ASR derived reference (speech fraction 91.9%): + +| System | Precision | Recall | F1 | Speech fraction found | +| --- | ---: | ---: | ---: | ---: | +| Silero M | 97.3 | 86.8 | 91.8 | 82.0% | +| Silero D | 97.4 | 87.4 | 92.1 | not recorded | +| Ultra Q8_0 | 97.1 | 90.7 | 93.8 | 85.8% | +| Redux packed | 97.1 | 91.5 | 94.2 | 86.6% | + +Agreement: Ultra and Silero M F1 97.0, kappa 0.813; Redux and Silero M F1 96.9, +kappa 0.801; Ultra and Redux F1 98.8, kappa 0.912. The "Silero D" speech fraction +was not printed in the run output. + +The ordering is the same as on the synthetic clips: equal precision, the heads +have the higher recall. The gap in F1 between Silero M and the heads is 2 to 2.5 +points. Because the reference comes from a Parakeet model, a bias in favor of the +Parakeet heads is possible and was not measured. + +### Results: speed of the heads and of Silero ONNX + +300 s of one talk (the first 300 s of the 1249 s talk), 4 repetitions, load average +5.8 to 6.2 during the run. Silero ran through the `silero-vad` Python package +(ONNX, one thread). The heads ran with 8 threads. + +| System | Speed, x real time | +| --- | ---: | +| Silero ONNX, 1 thread | 393 | +| Ultra Q8_0, 8 threads | 216 | +| Ultra F16, 8 threads | 218 | +| Redux packed, 8 threads | 221 | +| Redux dequantized F16, 8 threads | 222 | + +The run printed one speed per system. The result file does not say whether it is the +best or the median of the 4 repetitions (unverified). The speeds that the TED script +printed during the accuracy run (load average 6 to 42) are not valid and are not +used. Thread pinning was not recorded for this run. + +### Caveats + +- References are coarse. The synthetic reference counts pauses inside utterances as + speech and uses an energy trim. The TED reference comes from ASR word times. +- Absolute precision and recall depend on that reference. Compare detectors with + each other, on the same clips. +- Differences of less than about one F1 point between two detectors in one cell are + not claimed. There are no confidence intervals; one corpus was used. +- Speed numbers: see the load notes above. Detectors with different thread counts + are not a model comparison. +- The build commit of this run was not recorded. + +## Silero in whisper.cpp and in parakeet.cpp + +This benchmark asks whether the Silero port in parakeet.cpp gives the same result +and the same speed as the Silero VAD that ships in whisper.cpp. + +### Setup + +- **Models.** `silero_vad.onnx` (version 6.2.3, SHA-256 + `1a153a22f4509e292a94e67d6f9b85e8deb25b4988682b7e174c65279d8788e3`); + parakeet.cpp GGUF F32 (`1398e5ce...`) and F16 (`8160489282...`) made with + `scripts/convert_silero_vad_to_gguf.py` from that ONNX file; whisper.cpp model + `ggml-silero-v6.2.3-ggml.bin` (`abd79739...`), converted from the same ONNX with the + whisper.cpp conversion script, and the model of the whisper.cpp model repository + `ggml-silero-v6.2.0.bin` (`2aa269b7...`). The full list of hashes and versions is in + [`silero_versions_and_sha256.txt`](../scripts/vad_bench/results/silero_versions_and_sha256.txt). +- **Reference.** onnxruntime on the official ONNX file, with the official 64 sample + context, one thread. +- **whisper.cpp access.** A small test program (`scripts/vad_bench/wvad.cpp`) over the + public `whisper_vad_*` functions: probabilities, and the segments of the whisper.cpp + C++ code with its defaults and with matched parameters. It is not part of + whisper.cpp. +- **Corpus.** 120 synthetic clips (40 clips of 5 utterances, in clean, white 10 dB and + pink 10 dB), built with the same recipe as above from 200 LibriSpeech test-clean + utterances (every 13th). Plus the first 600 s of one TED-LIUM talk. +- **Segmentation.** To compare only the models, the same Python code turns every + probability stream into segments: threshold 0.5, negative threshold 0.35, minimum + speech 100 ms, minimum silence 200 ms, padding 0 ("matched post"). The own defaults + of whisper.cpp and parakeet.cpp are also shown. + +### Probability parity against onnxruntime + +All 120 clips and the 600 s talk, one value per 32 ms chunk: + +| System | Max abs diff | Mean abs diff | Frames > 0.05 off | Decision agreement at 0.5 | Flipped frames | +| --- | ---: | ---: | ---: | ---: | ---: | +| whisper.cpp, converted 6.2.3 | 0.7373 | 0.00224 | 0.54% | 99.826% | 309 | +| whisper.cpp, repository 6.2.0 | 0.7373 | 0.00224 | 0.54% | 99.826% | 309 | +| parakeet.cpp F32 | 0.0001 | 0.00002 | 0.00% | 99.999% | 2 | +| parakeet.cpp F16 | 0.0036 | 0.00005 | 0.00% | 99.997% | 5 | + +whisper.cpp (converted 6.2.3) against parakeet.cpp F32: mean abs diff 0.00224, 311 +flipped frames. The two whisper.cpp model files give the same result (0 flipped +frames between them). Mean abs diff against onnxruntime, first chunk of each clip / +the rest: whisper.cpp 0.00088 / 0.00212, parakeet.cpp F32 0.00002 / 0.00002, F16 +0.00005 / 0.00005. On the 600 s talk alone, the number of flipped frames against +onnxruntime is 57 of 18750 for whisper.cpp, 0 of 18750 for parakeet.cpp F32 and 1 +for F16. The cause of the whisper.cpp difference was not investigated. + +### Segment quality on the synthetic clips + +Precision / recall / F1 in percent, 40 clips per condition, same reference as above: + +| System (settings) | clean | white 10 dB | pink 10 dB | all | +| --- | --- | --- | --- | --- | +| onnxruntime (matched post) | 99.3 / 89.3 / 94.0 | 99.4 / 87.8 / 93.2 | 99.5 / 87.4 / 93.1 | 99.4 / 88.2 / 93.5 | +| whisper.cpp (matched post) | 99.3 / 89.4 / 94.1 | 99.4 / 87.8 / 93.3 | 99.5 / 87.4 / 93.0 | 99.4 / 88.2 / 93.5 | +| parakeet.cpp F32 (matched post) | 99.3 / 89.3 / 94.0 | 99.4 / 87.8 / 93.2 | 99.5 / 87.4 / 93.1 | 99.4 / 88.2 / 93.5 | +| parakeet.cpp F16 (matched post) | 99.3 / 89.3 / 94.0 | 99.4 / 87.8 / 93.2 | 99.5 / 87.4 / 93.1 | 99.4 / 88.2 / 93.5 | +| whisper.cpp (its own defaults) | 98.9 / 90.3 / 94.4 | 99.2 / 89.0 / 93.8 | 99.3 / 88.6 / 93.6 | 99.1 / 89.3 / 94.0 | +| whisper.cpp (its C++ code, matched parameters) | 99.3 / 89.4 / 94.1 | 99.4 / 87.8 / 93.3 | 99.5 / 87.4 / 93.0 | 99.4 / 88.2 / 93.5 | +| parakeet.cpp F32 (its own defaults) | 99.1 / 89.3 / 93.9 | 99.3 / 87.6 / 93.1 | 99.4 / 87.3 / 93.0 | 99.3 / 88.1 / 93.3 | + +Boundary error against the reference, all conditions pooled (start median / 90th +percentile, end median / 90th percentile, miss rate): onnxruntime, whisper.cpp, +parakeet.cpp F32 and F16 with matched post all give 241/553 and 108/396, 0.2%. +whisper.cpp with its own defaults gives 209/521, 124/367, 0.0%. parakeet.cpp with its +own defaults gives 211/523, 116/392, 0.0%. + +Agreement of the speech masks (matched post), 120 clips pooled (F1 / kappa / IoU): + +| Pair | F1 | kappa | IoU | +| --- | ---: | ---: | ---: | +| whisper.cpp vs parakeet.cpp F32 | 99.92 | 0.9969 | 99.84 | +| whisper.cpp vs onnxruntime | 99.92 | 0.9969 | 99.84 | +| parakeet.cpp F32 vs onnxruntime | 100.00 | 1.0000 | 100.00 | +| parakeet.cpp F16 vs F32 | 100.00 | 0.9999 | 100.00 | +| whisper.cpp 6.2.3 vs 6.2.0 file | 100.00 | 1.0000 | 100.00 | + +On the 600 s talk (no reference), speech fraction found: onnxruntime 79.1%, +whisper.cpp 78.8%, parakeet.cpp F32 79.1%. whisper.cpp against parakeet.cpp F32: +F1 99.75, kappa 0.9882. + +What this supports: with the same post processing, whisper.cpp Silero and parakeet.cpp +Silero give the same segment quality on these clips (all within 0.1 F1 point). The +difference between whisper.cpp and parakeet.cpp in their own defaults (94.0 against +93.3 F1 on all clips) comes from their different default settings, not from the model. +The F16 file is not measurably worse than F32 here. + +### Speed + +Each job ran on one pinned core, on a 300 s clip (the first 300 s of the 600 s talk; the +clip file was not kept, so this is assumed from the preparation script), +serialized with a lock. A run started only when the mean number of other runnable +tasks over 10 s was below 4, or the 1-minute load average was below 5. The machine has +20 logical CPUs and was never idle, so the 1-minute load average at the start of the +runs was between 4.2 and 15.9, and the mean runnable count was 1.8 to 5.5. These are +not quiet machine numbers. Wall time is the process wall time, including +process start and model load for the command line tools. + +| Job | Runs | Wall time, median (min to max), s | In process time, median, s | x real time (median) | Peak RSS | +| --- | ---: | ---: | ---: | ---: | ---: | +| whisper.cpp `detect_speech` only (test program) | 5 | 0.456 (0.446 to 0.473) | 0.438 | 686 | 36 MB | +| `whisper-vad-speech-segments` | 5 | 0.452 (0.444 to 0.457) | not applicable | 664 | 28 MB | +| `parakeet-cli vad`, F32 | 5 | 0.455 (0.425 to 0.464) | not applicable | 659 | 60 MB | +| `parakeet-cli vad`, F16 | 5 | 0.451 (0.436 to 0.553) | not applicable | 666 | 60 MB | +| onnxruntime, Python, 1 thread | 3 | 0.806 (0.738 to 1.241) | 0.668 | 449 | 104 MB | + +The x real time column is 300 s divided by the in process time when there is one, +else by the wall time. The four Silero command line jobs are within the spread of +their repeated runs (0.43 to 0.55 s), so the numbers do not show a speed difference +between whisper.cpp and parakeet.cpp. The onnxruntime figure is a Python loop that +calls the session once per 512 sample chunk. It is not a tuned onnxruntime number. + +whisper.cpp with `tiny.en`, one thread, same 300 s clip, end to end: 15.14 s +without VAD, 19.68 s with the Silero VAD (3 runs each, load average 4.6 to 8.3). The +ASR part dominates. Why the run with VAD is slower was not investigated. + +### Caveats + +- The load notes above apply. The runs were on a shared machine. +- One clip and one core. No claim is made about other lengths, other CPUs, or more + threads. +- The onnxruntime run had 3 repetitions. The other jobs had 5. +- The measured whisper.cpp is a source build at the commit above, CPU only. + +## Long talks: batched decode for VAD segments + +PR 82 decodes the VAD segments in groups of up to 16, using the exact batched +decoder of PR 81. Each segment is still encoded alone. This measures that change, not +the VAD itself. + +Data: four TED-LIUM long-form talks of 18 to 25 minutes, merged to one file each. +Models: Ultra Q8_0 and Redux packed. Command: `parakeet-cli transcribe --vad --json +--threads 8`, pinned to 8 cores. Three builds, interleaved, 5 runs each: +**120 runs**. The build order was rotated for each repetition. Before each run the +script waited until the mean number of other runnable tasks over 10 s was below 4. +The mean number of other runnable tasks at the gate was 0 to 1.5, but the 1-minute +load average at the start of the runs was 2.5 to 18.0 (it includes the earlier runs of +the same job and other jobs), see +`scripts/vad_bench/results/longform_b1_loadlog.txt` and the per run values in +`longform_b1_results.tsv`. The 1-minute load average was therefore often above 5. +Wall time is the time of the whole command, including model load. + +Median wall time in seconds over 5 runs: + +| Talk (length, s) | Model | Segments | base | PR 81 | PR 82 | PR 82 vs PR 81 | Peak RSS PR 82 vs PR 81 | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| BillGates (1505.8) | Ultra Q8_0 | 59 | 44.2 | 43.7 | 40.8 | 1.072x | +1.7% | +| BillGates | Redux packed | 62 | 28.1 | 27.7 | 25.9 | 1.071x | +2.2% | +| AimeeMullins (1249.0) | Ultra Q8_0 | 46 | 36.9 | 36.9 | 34.8 | 1.058x | +1.7% | +| AimeeMullins | Redux packed | 46 | 23.4 | 23.7 | 21.9 | 1.079x | +2.5% | +| JaneMcGonigal (1167.6) | Ultra Q8_0 | 50 | 35.9 | 34.9 | 34.0 | 1.029x | +1.4% | +| JaneMcGonigal | Redux packed | 47 | 24.1 | 23.1 | 23.0 | 1.006x | +2.2% | +| DanielKahneman (1095.5) | Ultra Q8_0 | 44 | 32.6 | 33.1 | 32.9 | 1.006x | +1.7% | +| DanielKahneman | Redux packed | 42 | 21.4 | 21.5 | 21.2 | 1.013x | +2.2% | + +The average of the eight speed ratios is 1.042x (range 1.006x to 1.079x). Peak +memory of the whole transcription is 1.4 to 2.5 percent higher with PR 82, 30 to 40 +MB on a base of 1.4 to 2.0 GB. The transcript (JSON output) has the same hash for all +three builds in all 8 talk and model pairs. + +Caveats: the gain is small and several of the ratios are inside the spread of the +5 runs (some single runs of the same configuration differ by more than 3%); do not +claim more than a small gain. Only CPU and x86 were measured. The memory numbers are +for the whole transcription, not for the VAD. + +## Silero parity with the reference (PR 83) + +This table is also in [vad.md](vad.md). Test clip: `tests/fixtures/speech.wav` padded +with silence (310 chunks), against onnxruntime 1.27.0 and the ONNX file of version +6.2.3. Maximum probability difference: + +| File | 16 kHz | 8 kHz | +| --- | ---: | ---: | +| F32 | 9e-7 | 2e-6 | +| F16 | 7e-4 | 3e-3 | + +One run of `tests/silero_vad_probe` on one CPU thread measured about 40 us per chunk +for F32 and F16 at both rates. That is one run on one machine. It is not a benchmark +and is not used above. + +## Slice only head (pending) + +Running the head needs the whole ASR GGUF today. A measurement of a head file that +holds only the head and the encoder part it needs ("slice") is being made by another +piece of work. It will give the load time, the peak memory and the speed of the head +as a stand alone detector. + +| Quantity | Value | +| --- | --- | +| Peak memory, head alone | pending | +| Load time, head alone | pending | +| Speed, head alone | pending | +| File size, head alone | pending | + +No number is given here until that measurement is published with its method. + +## Timing re-run + +Not run yet when this text was written. + +## When to use which + +This is limited to what the numbers above support. + +- **Stand alone gate, any ASR model, small files.** Silero. Its GGUF is 1.3 to 2.2 + MB, the command line tool needs about 60 MB, and it ran at about 660x real time on + one thread on a shared machine. It gave frame level F1 of 91.7 to 94.4 on the + synthetic data and 91.8 to 92.1 on the TED talks. +- **The Parakeet head when Ultra or Redux is already loaded.** The head costs no extra + model and its scores are close to Silero in the same tests: within about 1.5 F1 points + on the synthetic clips at 5 dB and above (Ultra was 0.4 lower in white noise at 5 dB), + 2 to 2.5 points higher on the TED talks, and 1.7 to 2.6 points higher at 0 dB pink + noise. It has the higher recall and the lower precision. On these clips its start times are closer to the reference. It + is not a cheap stand alone detector today: it needs the encoder, its speed is about + 220x on 8 threads, and its memory as a stand alone VAD is not yet measured. +- **Silero from whisper.cpp or from parakeet.cpp.** On the 120 clips and the 600 s talk + the two give the same segments (mask F1 99.92) and the same speed within the noise of + a shared machine. Choose by the rest of your stack, not by quality or speed. The + probabilities of parakeet.cpp match onnxruntime more closely (mean abs diff 0.00002, + 2 flipped frames) than those of whisper.cpp do (0.00224, 309 flipped frames), but this + did not change the segment scores. +- **Not supported by these numbers:** any claim about GPUs, ARM, other languages, + music or non speech noise, streaming latency, or the effect of the VAD on WER. + +## How to reproduce + +The scripts are in [`scripts/vad_bench/`](../scripts/vad_bench/), with a README that +lists the inputs, the commands and the Python packages. No audio and no model files +are committed. In short: + +1. Fetch the public audio: `fetch_data.py`, `fetch_ted_talks.py`. +2. Heads against Silero: `compare_synthetic.py` then `analyze_synthetic.py`; + `ted_reference.py` then `compare_ted.py`; `speed_heads.py`. +3. Silero parity: `make_clips.py`, build `wvad.cpp` against whisper.cpp, + `silero_collect.py`, `silero_analyze.py`; speed with `speed_silero.py`. +4. Long talks: `longform_b1.sh`. + +Small result files of the runs on this page (tables, per run timings and load logs) +are in `scripts/vad_bench/results/`. The raw per clip predictions are not committed; +a re-run regenerates them. diff --git a/docs/vad.md b/docs/vad.md index 0155674..a8eda25 100644 --- a/docs/vad.md +++ b/docs/vad.md @@ -12,6 +12,9 @@ Both give one speech probability per frame. A shared segmenter transcription segments. The segmenter takes the frame period as a parameter (`SegmenterOpts::frame_sec`); only the option defaults differ per detector. +Accuracy and speed numbers for both detectors, with the method and the scripts to +repeat them, are in [vad-benchmarks.md](vad-benchmarks.md). + ## Standalone VAD API One call returns the speech regions of a clip as JSON, for either detector. Load @@ -242,7 +245,7 @@ onnxruntime 1.27.0 and the ONNX file of version 6.2.3: On one CPU thread, `tests/silero_vad_probe` measured about 40 us per chunk (F32 or F16, 16 kHz or 8 kHz) on the development machine. That is one run on one -machine, not a benchmark. +machine, not a benchmark. For benchmarks, see [vad-benchmarks.md](vad-benchmarks.md). ## Tests diff --git a/scripts/vad_bench/README.md b/scripts/vad_bench/README.md new file mode 100644 index 0000000..dd89a34 --- /dev/null +++ b/scripts/vad_bench/README.md @@ -0,0 +1,69 @@ +# VAD benchmark scripts + +Scripts behind [docs/vad-benchmarks.md](../../docs/vad-benchmarks.md). The page says what +each number means and which caveats apply. No audio and no model files are committed; +`results/` holds the small result files of the runs on that page. + +All paths are arguments. Run from the repository root. `PARAKEET_LIB` is the path of +`libparakeet.so` (build with `-DPARAKEET_SHARED=ON`; the default is `build/libparakeet.so`). + +Python packages: `numpy soundfile librosa datasets torch onnxruntime silero-vad` +(the Silero reference is `silero-vad` 6.2.3). Models: the ASR GGUFs of +`moondream/parakeet-ultra` (Q8_0 and F16), `moondream/parakeet-redux` (packed `keep` and +dequantized F16), a TDT 0.6B v3 F16 GGUF for the reference of the TED talks, and the +Silero GGUFs made with `scripts/convert_silero_vad_to_gguf.py`. + +## Data + +| Script | What | +| --- | --- | +| `fetch_data.py OUT [--libri-count N] [--ted-seconds S]` | LibriSpeech test-clean (every 13th utterance) as `libri.npy`, and the first 600 s of one TED-LIUM talk | +| `fetch_ted_talks.py OUT [N]` | the first N TED-LIUM long-form talks under 1500 s as `ted_.wav` | +| `make_clips.py DATA` | the 120 synthetic clips (clean, white 10 dB, pink 10 dB) of the Silero parity run | +| `vad_synth.py` | clip builder, noise, and metrics (frame F1, kappa, boundary error) | + +## Parakeet heads against Silero + +``` +scripts/vad_bench/compare_synthetic.py DATA GGUF_DIR out/synthetic_raw.json 24 +scripts/vad_bench/analyze_synthetic.py out/synthetic_raw.json +scripts/vad_bench/ted_reference.py DATA tdt-0.6b-v3-f16.gguf # words of each talk +scripts/vad_bench/compare_ted.py DATA GGUF_DIR out/ted_metrics.json +taskset -c 0-7 scripts/vad_bench/speed_heads.py talk.wav GGUF_DIR # timing, quiet machine only +``` + +`GGUF_DIR` holds `ultra-q8_0.gguf`, `ultra-f16.gguf`, `redux-keep.gguf` and +`redux-deq-f16.gguf`. `compare_ted.py` prints speeds; ignore them, use `speed_heads.py`. + +## Silero in whisper.cpp and in parakeet.cpp + +Build whisper.cpp (CPU), then the test program, which uses its public `whisper_vad_*` API: + +``` +g++ -O2 -std=c++17 scripts/vad_bench/wvad.cpp -I whisper.cpp/include -I whisper.cpp/ggml/include \ + -L whisper.cpp/build/bin -lwhisper -Wl,-rpath,$PWD/whisper.cpp/build/bin -o wvad +python3 scripts/vad_bench/silero_collect.py CORE DATA MODEL_DIR build/examples/cli/parakeet-cli ./wvad +python3 scripts/vad_bench/silero_analyze.py DATA # run where probs.npz was written +python3 scripts/vad_bench/speed_silero.py CORE ROUNDS DATA/ted300.wav MODEL_DIR \ + build/examples/cli/parakeet-cli ./wvad whisper.cpp/build/bin out/speed.json +``` + +`ted300.wav` is the first 300 s of `ted_talk.wav`. `MODEL_DIR` holds `silero_vad.onnx`, +`silero-vad-f32.gguf`, `silero-vad-f16.gguf`, `ggml-silero-v6.2.3-ggml.bin`, +`ggml-silero-v6.2.0.bin` and, for the end to end job, `ggml-tiny.en.bin`. +`silero_analyze.py` expects `probs.npz` and `segs_own_default.json` in the current directory. + +## Long talks, batched decode + +``` +scripts/vad_bench/longform_b1.sh TALKS_DIR GGUF_DIR BUILDS_DIR out.tsv loadlog.txt +``` + +`BUILDS_DIR` has `build-base`, `build-b0`, `build-b1`: builds of the three commits to compare. + +## Timing rules + +Timing scripts serialize on a lock file (`/tmp/pk-bench.lock`, `flock`), pin cores with +`taskset`, and record the load average at the start of every run. Run them only on a +quiet machine, and keep the load log with the result. Do not quote a timing from a +loaded machine as a benchmark. diff --git a/scripts/vad_bench/analyze_synthetic.py b/scripts/vad_bench/analyze_synthetic.py new file mode 100755 index 0000000..25c6622 --- /dev/null +++ b/scripts/vad_bench/analyze_synthetic.py @@ -0,0 +1,62 @@ +#!/usr/bin/env python3 +"""usage: analyze_synthetic.py synthetic_raw.json -> tables on stdout""" +import json,sys,numpy as np +from heads_lib import mask,counts,prf,kappa,bound_err +R=json.load(open(sys.argv[1])) +conds=R["conds"];refs=R["refs"];P=R["preds"] +dur=lambda i: refs[i][-1][1]+1.0 # unused +def total_sec(i): + return None +import soundfile +# clip length: need exact; recompute from the raw: last pred... use max(ref end, pred end)+2 s margin rounding => use mask length from ref end + 1.5 (trailing silence >=0.5) +def n_sec(i): return max(refs[i][-1][1]+0.5, max([e for s,e in (P['silero_matched'][i] or [(0,0)])]+[0]))+0.0 +order=["clean","white20","white10","white5","white0","pink20","pink10","pink5","pink0"] +def agg(sysname,cs): + c=np.zeros(4);bs=[];be=[];ms=0;me=0;nb=0 + for i,cn in enumerate(conds): + if cn not in cs or P[sysname][i] is None: continue + L=n_sec(i)+3 + ref=mask(refs[i],L);pr=mask(P[sysname][i],L) + c+=counts(pr,ref) + e1,m1=bound_err(P[sysname][i],refs[i],0);e2,m2=bound_err(P[sysname][i],refs[i],1) + bs+=e1;be+=e2;ms+=m1;me+=m2;nb+=len(refs[i]) + return c,bs,be,ms,me,nb +out={} +systems=["silero_matched","silero_default","ultra_q8","redux_packed"] +print("FRAME-LEVEL (10 ms grid), thr 0.5") +print("cond | "+" | ".join(f"{s} P/R/F1" for s in systems)) +rows=[] +for cn in order: + cells=[] + for s in systems: + c,*_=agg(s,[cn]);p,r,f=prf(c);cells.append(f"{p*100:.1f}/{r*100:.1f}/{f*100:.1f}");out[f"{cn}|{s}"]=[p,r,f] + print(cn," | ".join(cells)) +print("\nBOUNDARY ERROR ms (median / p90 / miss%) start;end, pooled") +for cn in ["clean","white10","white0","pink10","pink0"]: + cells=[] + for s in systems: + c,bs,be,ms,me,nb=agg(s,[cn]) + f=lambda a:(f"{np.median(a):.0f}/{np.percentile(a,90):.0f}" if a else "-") + cells.append(f"S {f(bs)} ({100*ms/nb:.1f}%) E {f(be)} ({100*me/nb:.1f}%)") + print(cn," | ".join(cells)) +print("all-conditions boundary:") +for s in systems: + c,bs,be,ms,me,nb=agg(s,order); a=bs+be + print(s,f"median {np.median(a):.0f} p90 {np.percentile(a,90):.0f} start-med {np.median(bs):.0f} end-med {np.median(be):.0f} miss {100*(ms+me)/(2*nb):.1f}%") +print("\nTHRESHOLD SWEEP F1 (clean,white10,white0,pink10 pooled) P/R/F1") +sub=["clean","white10","white0","pink10"] +for base in ["silero_matched","ultra_q8","redux_packed"]: + cells=[] + for t in ("@0.3","","@0.7"): + nm=base+t; c,*_=agg(nm,sub); p,r,f=prf(c); cells.append(f"thr{t or '@0.5'} {p*100:.1f}/{r*100:.1f}/{f*100:.1f}") + print(base," | ".join(cells)) +print("\nAGREEMENT silero_matched vs parakeet (all synthetic): F1 of speech mask, kappa, IoU") +for s in ["ultra_q8","redux_packed"]: + c=np.zeros(4) + for i in range(len(conds)): + L=n_sec(i)+3;c+=counts(mask(P[s][i],L),mask(P["silero_matched"][i],L)) + p,r,f=prf(c);print(s,f"F1 {f*100:.1f} kappa {kappa(c):.3f} IoU {c[0]/(c[0]+c[1]+c[2])*100:.1f}") +c=np.zeros(4) +for i in range(len(conds)): + L=n_sec(i)+3;c+=counts(mask(P["ultra_q8"][i],L),mask(P["redux_packed"][i],L)) +print("ultra vs redux",f"F1 {prf(c)[2]*100:.1f} kappa {kappa(c):.3f}") diff --git a/scripts/vad_bench/compare_synthetic.py b/scripts/vad_bench/compare_synthetic.py new file mode 100755 index 0000000..4b44082 --- /dev/null +++ b/scripts/vad_bench/compare_synthetic.py @@ -0,0 +1,50 @@ +#!/usr/bin/env python3 +"""Synthetic benchmark: Parakeet VAD head (Ultra Q8_0, Redux packed) vs Silero. + +usage: PARAKEET_LIB=build/libparakeet.so compare_synthetic.py DATA_DIR GGUF_DIR OUT.json [CLIPS_PER_COND] +GGUF_DIR holds ultra-q8_0.gguf and redux-keep.gguf. Needs: numpy soundfile torch silero-vad onnxruntime. +Writes the raw spans of every system; analyze_synthetic.py turns them into the tables. +""" +import json,sys,numpy as np,soundfile as sf +from heads_lib import * +SYS={} +def make_systems(): + S={"silero_matched":lambda y:silero(y,True),"silero_default":lambda y:silero(y,False)} + return S +DATA,G,OUT=sys.argv[1],sys.argv[2],sys.argv[3] # data dir, gguf dir, output json +N_CLIPS=int(sys.argv[4]) if len(sys.argv)>4 else 24 # clips per condition +libri=list(np.load(DATA+"/libri.npy",allow_pickle=True)) +rng=np.random.default_rng(7) +order=rng.permutation(len(libri)) +sets=[[libri[i] for i in order[k*5:(k+1)*5]] for k in range(len(libri)//5)][:N_CLIPS] +conds=[("clean",None)]+[(k,s) for k in("white","pink") for s in(20,10,5,0)] +# build corpus once +corpus=[] +for ci,(kind,snr) in enumerate(conds): + for k,us in enumerate(sets): + r=np.random.default_rng(1000+k) # same speech layout per clip across conditions + y,spans=build(us,r) + y=add_noise(y,spans,kind,snr,np.random.default_rng(5000+ci*100+k)) + corpus.append((f"{kind}{'' if snr is None else snr}",y,spans)) +print("corpus",len(corpus),"wavs",sum(len(c[1]) for c in corpus)/16000,"s",flush=True) +models={"ultra_q8":"ultra-q8_0.gguf","redux_packed":"redux-keep.gguf"} +preds={} # system -> list of spans per corpus item +for name,fn in make_systems().items(): + preds[name]=[fn(c[1]) for c in corpus] +print("silero done",flush=True) +for mname,f in models.items(): + pk=PK(G+"/"+f) + preds[mname]=[parakeet(pk,c[1]) for c in corpus] + # threshold sweep on a subset of conditions + for thr in (0.3,0.7): + preds[f"{mname}@{thr}"]=[parakeet(pk,c[1],thr) if c[0] in("clean","white10","white0","pink10") else None for c in corpus] + print(mname,"done",flush=True) +# silero thresholds +for thr in (0.3,0.7): + preds[f"silero_matched@{thr}"]=[None]*len(corpus) + for i,c in enumerate(corpus): + if c[0] in("clean","white10","white0","pink10"): + ts=get_speech_timestamps(torch.from_numpy(c[1]),SIL,sampling_rate=16000,return_seconds=True,threshold=thr,min_speech_duration_ms=100,min_silence_duration_ms=200,speech_pad_ms=0) + preds[f"silero_matched@{thr}"][i]=[(t["start"],t["end"]) for t in ts] +json.dump({"conds":[c[0] for c in corpus],"refs":[c[2] for c in corpus],"preds":preds},open(OUT,"w")) +print("saved",flush=True) diff --git a/scripts/vad_bench/compare_ted.py b/scripts/vad_bench/compare_ted.py new file mode 100755 index 0000000..957f0c5 --- /dev/null +++ b/scripts/vad_bench/compare_ted.py @@ -0,0 +1,51 @@ +#!/usr/bin/env python3 +"""usage: compare_ted.py DATA_DIR GGUF_DIR OUT.json +DATA_DIR holds ted_*.wav and the matching ted_*_words.json from ted_reference.py. +The speed it prints is NOT a valid benchmark (other load); use speed_heads.py.""" +import json,glob,sys,time,subprocess,numpy as np,soundfile as sf +from heads_lib import * +from pk import PK +DATA,G,OUT=sys.argv[1],sys.argv[2],sys.argv[3] +print("uptime before:",subprocess.getoutput("uptime")) +talks=[f for f in sorted(glob.glob(DATA+"/ted_*.wav")) if "S103" not in f] # the original run left out one talk with S103 in its name; reason not recorded +ys={f:sf.read(f,dtype="float32")[0] for f in talks} +refs={} +for f in talks: + W=json.load(open(f.replace(".wav","_words.json"))) + spans=[]; + for s,e in W: + if spans and s-spans[-1][1]<0.4: spans[-1][1]=max(spans[-1][1],e) + else: spans.append([s,e]) + refs[f]=spans +res={};preds={} +def run(name,fn): + cs=np.zeros(4);tt=0;ds=0; + for f in talks: + y=ys[f];t0=time.perf_counter();p=fn(y);dt=time.perf_counter()-t0 + preds[(name,f)]=p;tt+=dt;ds+=len(y)/16000 + L=len(y)/16000+1;cs+=counts(mask(p,L),mask(refs[f],L)) + P,R,F=prf(cs);res[name]=dict(P=P,R=R,F1=F,rtf_x=ds/tt,sec=tt,audio=ds,speech_frac=None) + print(name,"P %.1f R %.1f F1 %.1f VAD speed %.0fx real time (%.1f s for %.0f s)"%(P*100,R*100,F*100,ds/tt,tt,ds),flush=True) +run("silero_matched",lambda y:silero(y,True)) +run("silero_default",lambda y:silero(y,False)) +for n,f in(("ultra_q8","ultra-q8_0.gguf"),("redux_packed","redux-keep.gguf")): + pk=PK(G+"/"+f); run(n,lambda y:parakeet(pk,y)); + out=pk.vad(ys[talks[1]],{"threshold":0.5}) + del pk +# reference speech fraction and agreement +cs=np.zeros(4) +for f in talks: + L=len(ys[f])/16000+1; r=mask(refs[f],L); cs+=np.array([r.sum(),0,0,len(r)-r.sum()]) +print("ref speech fraction %.1f%%"%(100*cs[0]/(cs[0]+cs[3]))) +for n in ["silero_matched","silero_default","ultra_q8","redux_packed"]: + t=sum(mask(preds[(n,f)],len(ys[f])/16000+1).sum() for f in talks);a=sum(len(mask([],len(ys[f])/16000+1)) for f in talks) + print(n,"speech fraction %.1f%%"%(100*t/a)) +print("AGREEMENT on TED talks") +for a,b in(("ultra_q8","silero_matched"),("redux_packed","silero_matched"),("ultra_q8","redux_packed")): + c=np.zeros(4) + for f in talks: + L=len(ys[f])/16000+1;c+=counts(mask(preds[(a,f)],L),mask(preds[(b,f)],L)) + print(a,"vs",b,"F1 %.1f kappa %.3f"%(prf(c)[2]*100,kappa(c))) +print("talks",[(f,round(len(ys[f])/16000)) for f in talks]) +json.dump(res,open(OUT,"w"),indent=1) +print("uptime after:",subprocess.getoutput("uptime")) diff --git a/scripts/vad_bench/fetch_data.py b/scripts/vad_bench/fetch_data.py new file mode 100755 index 0000000..cc4cd0b --- /dev/null +++ b/scripts/vad_bench/fetch_data.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python3 +"""Fetch the public audio used by the VAD benchmarks (streams from the Hugging Face hub). + +usage: fetch_data.py OUT_DIR [--libri-count N] [--ted-seconds S] + +Writes OUT_DIR/libri.npy (LibriSpeech test-clean, every 13th utterance, N of them) and +OUT_DIR/ted_talk.wav (the first TED-LIUM long-form talk longer than 700 s, cut to S seconds, +default 600). The silero parity run used N=200. The vad-compare run used a subset with 24 x 5 +utterances; the script that cut it was not saved, so that corpus is not byte-identical to the +one in the published tables (see docs/vad-benchmarks.md). +""" +import argparse, io, os +import numpy as np, soundfile as sf, librosa +from datasets import load_dataset, Audio + +ap = argparse.ArgumentParser() +ap.add_argument("out") +ap.add_argument("--libri-count", type=int, default=200) +ap.add_argument("--ted-seconds", type=int, default=600) +a = ap.parse_args() +os.makedirs(a.out, exist_ok=True) + +ds = load_dataset("openslr/librispeech_asr", "clean", split="test", streaming=True).cast_column("audio", Audio(decode=False)) +U = [] +for i, ex in enumerate(ds): + if i % 13: + continue + y, sr = sf.read(io.BytesIO(ex["audio"]["bytes"]), dtype="float32") + assert sr == 16000 + U.append(y) + if len(U) >= a.libri_count: + break +np.save(f"{a.out}/libri.npy", np.array(U, dtype=object), allow_pickle=True) +print("libri", len(U)) + +ds = load_dataset("distil-whisper/tedlium-long-form", split="test", streaming=True).cast_column("audio", Audio(decode=False)) +for ex in ds: + y, sr = sf.read(io.BytesIO(ex["audio"]["bytes"]), dtype="float32") + if y.ndim > 1: + y = y.mean(1) + if sr != 16000: + y = librosa.resample(y, orig_sr=sr, target_sr=16000) + if len(y) / 16000 < 700: + continue + sf.write(f"{a.out}/ted_talk.wav", y[: a.ted_seconds * 16000], 16000, subtype="PCM_16") + print("ted", len(y) / 16000) + break diff --git a/scripts/vad_bench/fetch_ted_talks.py b/scripts/vad_bench/fetch_ted_talks.py new file mode 100755 index 0000000..ac2b8dc --- /dev/null +++ b/scripts/vad_bench/fetch_ted_talks.py @@ -0,0 +1,18 @@ +#!/usr/bin/env python3 +"""usage: fetch_ted_talks.py OUT_DIR [N_TALKS] -> OUT_DIR/ted_.wav, the first N talks under 1500 s of +distil-whisper/tedlium-long-form (test). The published run used three talks; which ones is not recorded.""" +import io, re, sys, numpy as np, soundfile as sf, librosa +from datasets import load_dataset, Audio +D=sys.argv[1]; N=int(sys.argv[2]) if len(sys.argv)>2 else 3 +# TED-LIUM long-form: first talks +ds=load_dataset("distil-whisper/tedlium-long-form",split="test",streaming=True).cast_column("audio",Audio(decode=False)) +n=0 +for i,ex in enumerate(ds): + y,sr=sf.read(io.BytesIO(ex["audio"]["bytes"]),dtype="float32") + if y.ndim>1:y=y.mean(1) + if sr!=16000:y=librosa.resample(y,orig_sr=sr,target_sr=16000) + if len(y)/16000>1500: continue + name=re.sub(r"[^A-Za-z0-9_-]","",(ex["audio"]["path"] or f"talk{i}").split("/")[-1].rsplit(".",1)[0]) or f"talk{i}" + sf.write(f"{D}/ted_{name}.wav",y,16000,subtype="PCM_16") + print(name,len(y)/16000,flush=True); n+=1 + if n>=N: break diff --git a/scripts/vad_bench/heads_lib.py b/scripts/vad_bench/heads_lib.py new file mode 100755 index 0000000..dd9d6a3 --- /dev/null +++ b/scripts/vad_bench/heads_lib.py @@ -0,0 +1,60 @@ +"""Detector wrappers for the Parakeet-head vs Silero comparison. +Silero reference: the `silero-vad` pip package (6.2.3), ONNX, one thread. Parakeet: libparakeet via pk.py. +The synthetic-clip helpers are the same code as vad_synth.py (kept apart so each script runs on its own). +""" +import numpy as np, torch +from pk import PK +torch.set_num_threads(1) +from silero_vad import load_silero_vad, get_speech_timestamps +SIL=load_silero_vad(onnx=True) +GR=0.01 # mask grid 10 ms +def mask(spans,n_sec): + m=np.zeros(int(round(n_sec/GR)),bool) + for s,e in spans: + m[int(round(s/GR)):int(round(e/GR))]=True + return m +def silero(y,matched=True): + kw=dict(threshold=0.5,min_speech_duration_ms=100,min_silence_duration_ms=200,speech_pad_ms=0) if matched else {} + ts=get_speech_timestamps(torch.from_numpy(y),SIL,sampling_rate=16000,return_seconds=True,**kw) + return [(t["start"],t["end"]) for t in ts] +def parakeet(pk,y,thr=0.5): + return [(s["start"],s["end"]) for s in pk.vad(y,{"threshold":thr})["segments"]] +def trim(u): + fr=len(u)//160; r=np.sqrt((u[:fr*160].reshape(fr,160)**2).mean(1)) + idx=np.where(r>r.max()*0.01)[0] + return u[idx[0]*160:(idx[-1]+1)*160] +def pink(n,rng): + X=np.fft.rfft(rng.standard_normal(n)); f=np.arange(len(X)); f[0]=1 + y=np.fft.irfft(X/np.sqrt(f),n); return y/y.std() +def build(utts,rng): + g=lambda a,b: rng.uniform(a,b) + parts=[];spans=[];t=g(0.5,1.5) + parts.append(rng.standard_normal(int(t*16000))*1e-3) + for i,u in enumerate(utts): + u=trim(u); spans.append((t,t+len(u)/16000)); parts.append(u); t+=len(u)/16000 + gap=g(0.3,2.0) if i1.0: miss+=1 + else: out.append(d*1000) + return out,miss + diff --git a/scripts/vad_bench/longform_b1.sh b/scripts/vad_bench/longform_b1.sh new file mode 100755 index 0000000..31db519 --- /dev/null +++ b/scripts/vad_bench/longform_b1.sh @@ -0,0 +1,41 @@ +#!/bin/bash +# Batched VAD decode on long talks (PR 82): wall time and peak RSS of `transcribe --vad`. +# usage: longform_b1.sh TALKS_DIR GGUF_DIR BUILDS_DIR OUT_TSV LOADLOG [MAXWAIT_SECONDS] +# TALKS_DIR holds -merged.wav for the four talks named below (TED-LIUM long-form, merged) +# GGUF_DIR holds ultra-q8_0.gguf and redux-keep.gguf +# BUILDS_DIR holds build-base, build-b0, build-b1 (the three builds being compared) +# Runs 5 repetitions per (talk, model, build), build order rotated per repetition, 8 threads pinned +# to cores 0-7, serialized with a lock. Before each run it waits until the mean number of other +# runnable tasks (over 10 s) is below 4. The transcript hash is written to the TSV so equality of +# the outputs can be checked. +T=$1; G=$2; B=$3; R=$4; L=$5; MAXWAIT=${6:-10800}; WAITED=0 +OUTD=$(mktemp -d) +others() { t=0; for i in 1 2 3 4 5 6 7 8 9 10; do n=$(ps -eo stat,comm --no-headers | awk '$1 ~ /^R/ && $2!="ps" && $2!="awk"' | wc -l); t=$((t+n)); sleep 1; done; echo "scale=1; $t/10" | bc; } +wait_quiet() { + while true; do + l=$(cut -d' ' -f1 /proc/loadavg); o=$(others) + if awk "BEGIN{exit !($o < 4)}"; then echo "$(date +%H:%M:%S) ok runnable_mean $o loadavg1 $l" >> "$L"; return 0; fi + echo "$(date +%H:%M:%S) runnable_mean $o loadavg1 $l waiting" >> "$L" + if [ $WAITED -ge $MAXWAIT ]; then echo "GAVE UP after ${WAITED}s" >> "$L"; return 1; fi + sleep 240; WAITED=$((WAITED+250)) + done +} +for talk in BillGates AimeeMullins JaneMcGonigal DanielKahneman; do + for model in ultra-q8_0 redux-keep; do + for run in 1 2 3 4 5; do + builds=(base b0 b1); k=$(( (run-1) % 3 )); builds=("${builds[@]:k}" "${builds[@]:0:k}") + for b in "${builds[@]}"; do + grep -q -P "^$talk\t$model\t$run\t$b\t" "$R" 2>/dev/null && continue + wait_quiet || exit 1 + lb=$(cut -d' ' -f1-3 /proc/loadavg) + flock /tmp/pk-bench.lock taskset -c 0-7 /usr/bin/time -f "%e %M" -o "$OUTD/tm.txt" \ + "$B/build-$b/examples/cli/parakeet-cli" transcribe --model "$G/$model.gguf" --input "$T/$talk-merged.wav" --vad --json --threads 8 > "$OUTD/o.json" 2> "$OUTD/err.txt" + rc=$? + la=$(cut -d' ' -f1-3 /proc/loadavg) + sha=$(sha256sum < "$OUTD/o.json" | cut -c1-16) + echo -e "$talk\t$model\t$run\t$b\t$(tail -1 "$OUTD/tm.txt")\t$sha\trc$rc\tload_before $lb\tload_after $la" >> "$R" + done + done + done +done +rm -rf "$OUTD"; echo ALLDONE >> "$L" diff --git a/scripts/vad_bench/make_clips.py b/scripts/vad_bench/make_clips.py new file mode 100755 index 0000000..8aecc45 --- /dev/null +++ b/scripts/vad_bench/make_clips.py @@ -0,0 +1,31 @@ +#!/usr/bin/env python3 +"""Build the synthetic clips of the Silero parity benchmark. + +usage: make_clips.py DATA_DIR (DATA_DIR holds libri.npy from fetch_data.py) + +Writes DATA_DIR/clips/*.wav and DATA_DIR/clips.json: 40 clips of 5 utterances in 3 conditions +(clean, white noise at 10 dB SNR, pink noise at 10 dB SNR) = 120 clips. The reference speech +spans are the utterance spans (trimmed at 1 percent of peak frame RMS), so they are coarse. +""" +import json, os, sys +import numpy as np, soundfile as sf +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from vad_synth import build, add_noise + +D = sys.argv[1] +os.makedirs(D + "/clips", exist_ok=True) +libri = list(np.load(D + "/libri.npy", allow_pickle=True)) +rng = np.random.default_rng(7) +order = rng.permutation(len(libri)) +sets = [[libri[i] for i in order[k * 5:(k + 1) * 5]] for k in range(len(libri) // 5)] +conds = [("clean", None), ("white", 10), ("pink", 10)] +meta = [] +for ci, (kind, snr) in enumerate(conds): + for k, us in enumerate(sets): + y, spans = build(us, np.random.default_rng(1000 + k)) + y = add_noise(y, spans, kind, snr, np.random.default_rng(5000 + ci * 100 + k)) + name = f"{kind}{'' if snr is None else snr}_{k:02d}" + sf.write(f"{D}/clips/{name}.wav", y, 16000, subtype="PCM_16") + meta.append(dict(name=name, cond=name.split("_")[0], ref=spans, dur=len(y) / 16000)) +json.dump(meta, open(D + "/clips.json", "w")) +print(len(meta), "clips", sum(m["dur"] for m in meta), "s") diff --git a/scripts/vad_bench/onnx_time.py b/scripts/vad_bench/onnx_time.py new file mode 100755 index 0000000..2c1c97d --- /dev/null +++ b/scripts/vad_bench/onnx_time.py @@ -0,0 +1,10 @@ +import sys,time,numpy as np,soundfile as sf,onnxruntime as ort +so=ort.SessionOptions(); so.intra_op_num_threads=1; so.inter_op_num_threads=1 +s=ort.InferenceSession(sys.argv[1],so,providers=["CPUExecutionProvider"]) +y=sf.read(sys.argv[2],dtype="float32")[0]; n=(len(y)+511)//512; y=np.pad(y,(0,n*512-len(y))) +st=np.zeros((2,1,128),np.float32); ctx=np.zeros(64,np.float32); sr=np.array(16000,np.int64) +t0=time.perf_counter() +for i in range(n): + x=np.concatenate([ctx,y[i*512:(i+1)*512]])[None] + o,st=s.run(None,{"input":x,"state":st,"sr":sr}); ctx=x[0,-64:] +print("%.4f"%(time.perf_counter()-t0)) diff --git a/scripts/vad_bench/pk.py b/scripts/vad_bench/pk.py new file mode 100755 index 0000000..9ff0181 --- /dev/null +++ b/scripts/vad_bench/pk.py @@ -0,0 +1,26 @@ +"""ctypes wrapper over libparakeet for the VAD benchmarks. Set PARAKEET_LIB to the path of libparakeet.so.""" +import ctypes, json, os, numpy as np +LIB=os.environ.get("PARAKEET_LIB","build/libparakeet.so") +class PK: + def __init__(self, gguf): + L=self.L=ctypes.CDLL(LIB) + L.parakeet_capi_load.restype=ctypes.c_void_p; L.parakeet_capi_load.argtypes=[ctypes.c_char_p] + for n in ("parakeet_capi_vad_pcm_json","parakeet_capi_vad_path_json","parakeet_capi_transcribe_pcm_batch_json"): + getattr(L,n).restype=ctypes.c_void_p + L.parakeet_capi_vad_pcm_json.argtypes=[ctypes.c_void_p,ctypes.c_void_p,ctypes.c_int,ctypes.c_int,ctypes.c_char_p] + L.parakeet_capi_vad_path_json.argtypes=[ctypes.c_void_p,ctypes.c_char_p,ctypes.c_char_p] + L.parakeet_capi_transcribe_pcm_batch_json.argtypes=[ctypes.c_void_p,ctypes.c_void_p,ctypes.c_void_p,ctypes.c_int,ctypes.c_int,ctypes.c_int] + L.parakeet_capi_free_string.argtypes=[ctypes.c_void_p] + L.parakeet_capi_last_error.restype=ctypes.c_char_p; L.parakeet_capi_last_error.argtypes=[ctypes.c_void_p] + self.ctx=L.parakeet_capi_load(gguf.encode()); assert self.ctx + def _s(self,p): + if not p: raise RuntimeError(self.L.parakeet_capi_last_error(self.ctx).decode()) + s=ctypes.string_at(p).decode(); self.L.parakeet_capi_free_string(p); return s + def vad(self,pcm,opts=None): + pcm=np.ascontiguousarray(pcm,dtype=np.float32) + o=json.dumps(opts).encode() if opts else None + return json.loads(self._s(self.L.parakeet_capi_vad_pcm_json(self.ctx,pcm.ctypes.data,len(pcm),16000,o))) + def words_batch(self,clips): + cat=np.ascontiguousarray(np.concatenate(clips),dtype=np.float32) + n=np.array([len(c) for c in clips],dtype=np.int32) + return json.loads(self._s(self.L.parakeet_capi_transcribe_pcm_batch_json(self.ctx,cat.ctypes.data,n.ctypes.data,len(clips),16000,2))) diff --git a/scripts/vad_bench/results/heads_synthetic_signed.txt b/scripts/vad_bench/results/heads_synthetic_signed.txt new file mode 100644 index 0000000..320cf86 --- /dev/null +++ b/scripts/vad_bench/results/heads_synthetic_signed.txt @@ -0,0 +1,11 @@ +silero_matched start signed median 237 ms, p10 2 p90 548 +silero_matched end signed median 3 ms, p10 -404 p90 124 +silero_default start signed median 232 ms, p10 -46 p90 523 +silero_default end signed median 20 ms, p10 -370 p90 153 +ultra_q8 start signed median 162 ms, p10 -86 p90 469 +ultra_q8 end signed median -10 ms, p10 -420 p90 163 +redux_packed start signed median 147 ms, p10 -91 p90 459 +redux_packed end signed median 54 ms, p10 -332 p90 183 +silero_matched segments per ref span: 1.96 +ultra_q8 segments per ref span: 1.92 +redux_packed segments per ref span: 1.76 diff --git a/scripts/vad_bench/results/heads_synthetic_tables.txt b/scripts/vad_bench/results/heads_synthetic_tables.txt new file mode 100644 index 0000000..17cd523 --- /dev/null +++ b/scripts/vad_bench/results/heads_synthetic_tables.txt @@ -0,0 +1,33 @@ +FRAME-LEVEL (10 ms grid), thr 0.5 +cond | silero_matched P/R/F1 | silero_default P/R/F1 | ultra_q8 P/R/F1 | redux_packed P/R/F1 +clean 99.1/89.7/94.1 | 98.7/90.5/94.4 | 98.2/92.0/95.0 | 98.4/92.7/95.4 +white20 99.4/88.4/93.5 | 99.0/89.2/93.9 | 98.8/89.9/94.1 | 98.7/91.7/95.1 +white10 99.3/88.0/93.3 | 98.9/89.0/93.7 | 98.9/88.3/93.3 | 98.2/91.9/94.9 +white5 99.2/87.8/93.1 | 98.8/88.5/93.4 | 98.9/87.2/92.7 | 97.4/91.9/94.6 +white0 99.2/85.8/92.0 | 98.8/86.7/92.4 | 98.3/87.9/92.8 | 96.4/92.3/94.3 +pink20 99.4/88.4/93.6 | 99.1/89.3/93.9 | 98.8/90.8/94.6 | 98.6/92.0/95.2 +pink10 99.3/88.0/93.3 | 99.0/89.0/93.7 | 99.0/88.8/93.6 | 98.1/92.2/95.1 +pink5 99.4/87.6/93.1 | 99.0/88.5/93.5 | 99.0/88.4/93.4 | 97.4/92.6/94.9 +pink0 99.2/85.2/91.7 | 98.8/85.9/91.9 | 96.3/90.8/93.4 | 95.5/93.2/94.3 + +BOUNDARY ERROR ms (median / p90 / miss%) start;end, pooled +clean S 171/527 (0.0%) E 121/366 (0.0%) | S 162/499 (0.0%) E 137/343 (0.0%) | S 123/460 (0.0%) E 168/325 (0.0%) | S 152/459 (0.0%) E 151/328 (0.0%) +white10 S 249/526 (0.0%) E 116/371 (0.0%) | S 232/500 (0.0%) E 129/357 (0.0%) | S 211/470 (0.0%) E 147/472 (0.8%) | S 163/466 (0.8%) E 152/389 (0.8%) +white0 S 262/571 (0.8%) E 117/459 (2.5%) | S 237/526 (0.8%) E 142/451 (1.7%) | S 162/461 (0.0%) E 149/494 (2.5%) | S 202/460 (0.8%) E 149/404 (1.7%) +pink10 S 248/526 (0.0%) E 106/384 (0.0%) | S 232/502 (0.0%) E 117/343 (0.0%) | S 226/479 (0.0%) E 136/445 (0.0%) | S 176/445 (0.0%) E 148/340 (0.0%) +pink0 S 261/570 (0.0%) E 145/411 (3.3%) | S 248/530 (0.0%) E 150/408 (3.3%) | S 180/457 (2.5%) E 162/470 (2.5%) | S 200/457 (1.7%) E 160/428 (1.7%) +all-conditions boundary: +silero_matched median 143 p90 501 start-med 237 end-med 116 miss 0.4% +silero_default median 149 p90 484 start-med 232 end-med 130 miss 0.3% +ultra_q8 median 154 p90 465 start-med 181 end-med 149 miss 0.6% +redux_packed median 153 p90 437 start-med 170 end-med 149 miss 0.4% + +THRESHOLD SWEEP F1 (clean,white10,white0,pink10 pooled) P/R/F1 +silero_matched thr@0.3 98.5/90.3/94.3 | thr@0.5 99.2/87.9/93.2 | thr@0.7 99.5/86.2/92.4 +ultra_q8 thr@0.3 97.1/92.4/94.7 | thr@0.5 98.6/89.3/93.7 | thr@0.7 99.0/86.2/92.1 +redux_packed thr@0.3 94.8/95.0/94.9 | thr@0.5 97.7/92.3/94.9 | thr@0.7 98.9/90.0/94.2 + +AGREEMENT silero_matched vs parakeet (all synthetic): F1 of speech mask, kappa, IoU +ultra_q8 F1 96.6 kappa 0.882 IoU 93.4 +redux_packed F1 96.3 kappa 0.865 IoU 92.8 +ultra vs redux F1 97.2 kappa 0.896 diff --git a/scripts/vad_bench/results/heads_ted_and_speed_output.txt b/scripts/vad_bench/results/heads_ted_and_speed_output.txt new file mode 100644 index 0000000..4508bcb --- /dev/null +++ b/scripts/vad_bench/results/heads_ted_and_speed_output.txt @@ -0,0 +1,8 @@ +(console output of ted.py, load 6 to 42 during the run, speed numbers there are NOT valid; see speed.py output in the report) +silero_matched P 97.3 R 86.8 F1 91.8 +silero_default P 97.4 R 87.4 F1 92.1 +ultra_q8 P 97.1 R 90.7 F1 93.8 +redux_packed P 97.1 R 91.5 F1 94.2 +ref speech fraction 91.9%; speech fraction silero 82.0 / ultra 85.8 / redux 86.6 +agreement ultra-silero F1 97.0 kappa 0.813; redux-silero F1 96.9 kappa 0.801; ultra-redux F1 98.8 kappa 0.912 +speed.py (300 s of a talk, 4 reps, load 5.8-6.2): silero onnx 1 thr 393x; ultra_q8 216x; ultra_f16 218x; redux_packed 221x; redux_deq_f16 222x (8 threads) diff --git a/scripts/vad_bench/results/longform_b1_loadlog.txt b/scripts/vad_bench/results/longform_b1_loadlog.txt new file mode 100644 index 0000000..21f24b5 --- /dev/null +++ b/scripts/vad_bench/results/longform_b1_loadlog.txt @@ -0,0 +1,127 @@ +11:06:23 load1 12.88 waiting +11:11:23 load1 6.24 waiting +11:16:23 load1 6.73 waiting +11:21:23 load1 16.39 waiting +11:26:23 load1 8.36 waiting +11:32:09 load1 5.57 waiting +11:32:56 ok runnable_mean .1 loadavg1 4.07 (loadavg1 includes our own previous runs) +11:33:50 ok runnable_mean 0 loadavg1 7.06 (loadavg1 includes our own previous runs) +11:34:41 ok runnable_mean 0 loadavg1 8.94 (loadavg1 includes our own previous runs) +11:35:35 ok runnable_mean 0 loadavg1 9.07 (loadavg1 includes our own previous runs) +11:36:25 ok runnable_mean 0 loadavg1 8.73 (loadavg1 includes our own previous runs) +11:37:19 ok runnable_mean 0 loadavg1 8.72 (loadavg1 includes our own previous runs) +11:38:10 ok runnable_mean .1 loadavg1 8.26 (loadavg1 includes our own previous runs) +11:39:05 ok runnable_mean 0 loadavg1 7.93 (loadavg1 includes our own previous runs) +11:39:59 ok runnable_mean 0 loadavg1 8.50 (loadavg1 includes our own previous runs) +11:40:53 ok runnable_mean 0 loadavg1 9.20 (loadavg1 includes our own previous runs) +11:41:47 ok runnable_mean 0 loadavg1 8.62 (loadavg1 includes our own previous runs) +11:42:38 ok runnable_mean .1 loadavg1 7.63 (loadavg1 includes our own previous runs) +11:43:32 ok runnable_mean 0 loadavg1 8.26 (loadavg1 includes our own previous runs) +11:44:25 ok runnable_mean 0 loadavg1 7.83 (loadavg1 includes our own previous runs) +11:45:19 ok runnable_mean .1 loadavg1 8.19 (loadavg1 includes our own previous runs) +11:45:57 ok runnable_mean 0 loadavg1 8.16 (loadavg1 includes our own previous runs) +11:46:36 ok runnable_mean 0 loadavg1 8.47 (loadavg1 includes our own previous runs) +11:47:11 ok runnable_mean 0 loadavg1 7.84 (loadavg1 includes our own previous runs) +11:47:49 ok runnable_mean .1 loadavg1 8.15 (loadavg1 includes our own previous runs) +11:48:25 ok runnable_mean .1 loadavg1 7.90 (loadavg1 includes our own previous runs) +11:49:03 ok runnable_mean .1 loadavg1 8.71 (loadavg1 includes our own previous runs) +11:49:38 ok runnable_mean 0 loadavg1 8.12 (loadavg1 includes our own previous runs) +11:50:15 ok runnable_mean 0 loadavg1 7.81 (loadavg1 includes our own previous runs) +11:50:53 ok runnable_mean .1 loadavg1 8.60 (loadavg1 includes our own previous runs) +11:51:31 ok runnable_mean .1 loadavg1 9.13 (loadavg1 includes our own previous runs) +11:52:09 ok runnable_mean 0 loadavg1 8.25 (loadavg1 includes our own previous runs) +11:52:45 ok runnable_mean .1 loadavg1 7.47 (loadavg1 includes our own previous runs) +11:53:23 ok runnable_mean .1 loadavg1 7.70 (loadavg1 includes our own previous runs) +11:54:00 ok runnable_mean .1 loadavg1 7.57 (loadavg1 includes our own previous runs) +11:54:38 ok runnable_mean 0 loadavg1 7.11 (loadavg1 includes our own previous runs) +11:55:25 ok runnable_mean 0 loadavg1 7.75 (loadavg1 includes our own previous runs) +11:56:12 ok runnable_mean .2 loadavg1 8.74 (loadavg1 includes our own previous runs) +11:56:56 ok runnable_mean .3 loadavg1 8.60 (loadavg1 includes our own previous runs) +11:57:43 ok runnable_mean 0 loadavg1 8.07 (loadavg1 includes our own previous runs) +11:58:28 ok runnable_mean 0 loadavg1 8.62 (loadavg1 includes our own previous runs) +11:59:15 ok runnable_mean .1 loadavg1 8.62 (loadavg1 includes our own previous runs) +11:59:59 ok runnable_mean 0 loadavg1 8.55 (loadavg1 includes our own previous runs) +12:00:48 ok runnable_mean 0 loadavg1 8.71 (loadavg1 includes our own previous runs) +12:01:39 ok runnable_mean 0 loadavg1 8.99 (loadavg1 includes our own previous runs) +12:02:33 ok runnable_mean .2 loadavg1 9.98 (loadavg1 includes our own previous runs) +12:03:20 ok runnable_mean 0 loadavg1 9.46 (loadavg1 includes our own previous runs) +12:04:05 ok runnable_mean .2 loadavg1 8.98 (loadavg1 includes our own previous runs) +12:04:53 ok runnable_mean 0 loadavg1 8.86 (loadavg1 includes our own previous runs) +12:05:39 ok runnable_mean .8 loadavg1 9.07 (loadavg1 includes our own previous runs) +12:06:27 ok runnable_mean 0 loadavg1 8.74 (loadavg1 includes our own previous runs) +12:07:00 ok runnable_mean 0 loadavg1 8.65 (loadavg1 includes our own previous runs) +12:07:34 ok runnable_mean 0 loadavg1 8.55 (loadavg1 includes our own previous runs) +12:08:06 ok runnable_mean 0 loadavg1 8.94 (loadavg1 includes our own previous runs) +12:08:40 ok runnable_mean 0 loadavg1 8.00 (loadavg1 includes our own previous runs) +12:09:13 ok runnable_mean .1 loadavg1 7.60 (loadavg1 includes our own previous runs) +12:09:46 ok runnable_mean .1 loadavg1 7.28 (loadavg1 includes our own previous runs) +12:10:18 ok runnable_mean .1 loadavg1 7.19 (loadavg1 includes our own previous runs) +12:10:52 ok runnable_mean 0 loadavg1 7.59 (loadavg1 includes our own previous runs) +12:11:26 ok runnable_mean 0 loadavg1 8.07 (loadavg1 includes our own previous runs) +12:11:59 ok runnable_mean .1 loadavg1 8.06 (loadavg1 includes our own previous runs) +12:12:32 ok runnable_mean .1 loadavg1 8.03 (loadavg1 includes our own previous runs) +12:13:04 ok runnable_mean .1 loadavg1 8.12 (loadavg1 includes our own previous runs) +12:13:38 ok runnable_mean 0 loadavg1 7.93 (loadavg1 includes our own previous runs) +12:14:11 ok runnable_mean 1.5 loadavg1 8.46 (loadavg1 includes our own previous runs) +12:15:02 runnable_mean 11.3 loadavg1 17.44 waiting +12:19:13 ok runnable_mean .7 loadavg1 3.35 (loadavg1 includes our own previous runs) +12:19:58 ok runnable_mean 1.2 loadavg1 7.84 (loadavg1 includes our own previous runs) +12:20:44 ok runnable_mean .2 loadavg1 8.66 (loadavg1 includes our own previous runs) +12:21:27 ok runnable_mean .2 loadavg1 8.30 (loadavg1 includes our own previous runs) +12:22:13 ok runnable_mean 0 loadavg1 7.92 (loadavg1 includes our own previous runs) +12:22:57 ok runnable_mean .6 loadavg1 7.92 (loadavg1 includes our own previous runs) +12:23:43 ok runnable_mean .1 loadavg1 9.26 (loadavg1 includes our own previous runs) +12:24:27 ok runnable_mean 0 loadavg1 9.19 (loadavg1 includes our own previous runs) +12:25:13 ok runnable_mean 0 loadavg1 8.77 (loadavg1 includes our own previous runs) +12:25:58 ok runnable_mean .3 loadavg1 8.46 (loadavg1 includes our own previous runs) +12:26:44 ok runnable_mean 0 loadavg1 8.66 (loadavg1 includes our own previous runs) +12:27:30 ok runnable_mean .4 loadavg1 8.67 (loadavg1 includes our own previous runs) +12:28:13 ok runnable_mean .1 loadavg1 8.99 (loadavg1 includes our own previous runs) +12:28:58 ok runnable_mean .1 loadavg1 8.40 (loadavg1 includes our own previous runs) +12:29:44 ok runnable_mean .1 loadavg1 8.50 (loadavg1 includes our own previous runs) +12:30:31 ok runnable_mean .1 loadavg1 9.08 (loadavg1 includes our own previous runs) +12:31:06 ok runnable_mean .1 loadavg1 8.62 (loadavg1 includes our own previous runs) +12:31:40 ok runnable_mean .1 loadavg1 8.33 (loadavg1 includes our own previous runs) +12:32:16 ok runnable_mean .3 loadavg1 8.24 (loadavg1 includes our own previous runs) +12:32:51 ok runnable_mean .4 loadavg1 8.10 (loadavg1 includes our own previous runs) +12:33:41 ok runnable_mean 0 loadavg1 9.59 (loadavg1 includes our own previous runs) +12:34:16 ok runnable_mean .1 loadavg1 9.26 (loadavg1 includes our own previous runs) +12:34:49 ok runnable_mean 0 loadavg1 9.09 (loadavg1 includes our own previous runs) +12:35:24 ok runnable_mean .5 loadavg1 11.79 (loadavg1 includes our own previous runs) +12:35:57 ok runnable_mean .3 loadavg1 10.43 (loadavg1 includes our own previous runs) +12:36:30 ok runnable_mean .4 loadavg1 9.94 (loadavg1 includes our own previous runs) +12:37:03 ok runnable_mean .1 loadavg1 9.55 (loadavg1 includes our own previous runs) +12:37:33 ok runnable_mean 0 loadavg1 8.41 (loadavg1 includes our own previous runs) +12:38:06 ok runnable_mean .4 loadavg1 8.41 (loadavg1 includes our own previous runs) +12:38:37 ok runnable_mean .8 loadavg1 9.80 (loadavg1 includes our own previous runs) +12:39:11 ok runnable_mean .7 loadavg1 11.22 (loadavg1 includes our own previous runs) +12:39:53 ok runnable_mean .3 loadavg1 10.11 (loadavg1 includes our own previous runs) +12:40:39 ok runnable_mean 0 loadavg1 11.12 (loadavg1 includes our own previous runs) +12:41:23 ok runnable_mean .2 loadavg1 11.19 (loadavg1 includes our own previous runs) +12:42:06 ok runnable_mean 0 loadavg1 10.11 (loadavg1 includes our own previous runs) +12:42:51 ok runnable_mean 0 loadavg1 10.49 (loadavg1 includes our own previous runs) +12:43:37 ok runnable_mean .2 loadavg1 11.10 (loadavg1 includes our own previous runs) +12:44:20 ok runnable_mean .8 loadavg1 11.09 (loadavg1 includes our own previous runs) +12:45:04 ok runnable_mean .7 loadavg1 11.22 (loadavg1 includes our own previous runs) +12:45:47 ok runnable_mean .1 loadavg1 10.74 (loadavg1 includes our own previous runs) +12:46:30 ok runnable_mean .1 loadavg1 10.03 (loadavg1 includes our own previous runs) +12:47:13 ok runnable_mean .1 loadavg1 9.77 (loadavg1 includes our own previous runs) +12:47:55 ok runnable_mean 0 loadavg1 9.08 (loadavg1 includes our own previous runs) +12:48:38 ok runnable_mean .1 loadavg1 9.35 (loadavg1 includes our own previous runs) +12:49:19 ok runnable_mean 0 loadavg1 9.11 (loadavg1 includes our own previous runs) +12:50:02 ok runnable_mean 0 loadavg1 8.97 (loadavg1 includes our own previous runs) +12:50:32 ok runnable_mean .3 loadavg1 8.62 (loadavg1 includes our own previous runs) +12:51:04 ok runnable_mean .4 loadavg1 9.29 (loadavg1 includes our own previous runs) +12:51:35 ok runnable_mean 1.1 loadavg1 9.50 (loadavg1 includes our own previous runs) +12:52:07 ok runnable_mean .4 loadavg1 10.83 (loadavg1 includes our own previous runs) +12:52:46 ok runnable_mean 1.0 loadavg1 18.00 (loadavg1 includes our own previous runs) +12:53:20 ok runnable_mean 1.1 loadavg1 17.59 (loadavg1 includes our own previous runs) +12:53:51 ok runnable_mean .1 loadavg1 14.59 (loadavg1 includes our own previous runs) +12:54:22 ok runnable_mean 1.1 loadavg1 11.88 (loadavg1 includes our own previous runs) +12:54:57 ok runnable_mean .1 loadavg1 12.34 (loadavg1 includes our own previous runs) +12:55:27 ok runnable_mean 0 loadavg1 10.23 (loadavg1 includes our own previous runs) +12:55:57 ok runnable_mean .3 loadavg1 9.00 (loadavg1 includes our own previous runs) +12:56:26 ok runnable_mean .1 loadavg1 9.82 (loadavg1 includes our own previous runs) +12:56:58 ok runnable_mean .2 loadavg1 9.08 (loadavg1 includes our own previous runs) +12:57:33 ok runnable_mean 1.1 loadavg1 9.67 (loadavg1 includes our own previous runs) +ALLDONE diff --git a/scripts/vad_bench/results/longform_b1_results.tsv b/scripts/vad_bench/results/longform_b1_results.tsv new file mode 100644 index 0000000..6ba132b --- /dev/null +++ b/scripts/vad_bench/results/longform_b1_results.tsv @@ -0,0 +1,120 @@ +BillGates ultra-q8_0 1 base 45.96 2053092 69a20c09ad148e2f rc0 load_before 2.49 7.32 11.31 load_after 5.57 7.42 11.15 +BillGates ultra-q8_0 1 b0 43.41 2053224 69a20c09ad148e2f rc0 load_before 3.67 6.67 10.72 load_after 7.06 7.16 10.69 +BillGates ultra-q8_0 1 b1 41.24 2087760 69a20c09ad148e2f rc0 load_before 6.29 6.99 10.60 load_after 8.94 7.57 10.64 +BillGates ultra-q8_0 2 b0 43.74 2053060 69a20c09ad148e2f rc0 load_before 8.03 7.42 10.56 load_after 9.07 7.77 10.53 +BillGates ultra-q8_0 2 b1 39.54 2087804 69a20c09ad148e2f rc0 load_before 8.06 7.60 10.44 load_after 8.73 7.80 10.38 +BillGates ultra-q8_0 2 base 44.17 2053448 69a20c09ad148e2f rc0 load_before 7.70 7.61 10.29 load_after 8.72 7.90 10.26 +BillGates ultra-q8_0 3 b1 40.81 2088132 69a20c09ad148e2f rc0 load_before 7.69 7.70 10.17 load_after 8.26 7.80 10.10 +BillGates ultra-q8_0 3 base 44.72 2053220 69a20c09ad148e2f rc0 load_before 7.22 7.59 10.00 load_after 7.93 7.70 9.92 +BillGates ultra-q8_0 3 b0 44.35 2053268 69a20c09ad148e2f rc0 load_before 6.86 7.48 9.83 load_after 8.50 7.82 9.83 +BillGates ultra-q8_0 4 base 43.47 2053460 69a20c09ad148e2f rc0 load_before 7.81 7.70 9.77 load_after 9.20 8.10 9.81 +BillGates ultra-q8_0 4 b0 43.73 2053512 69a20c09ad148e2f rc0 load_before 7.94 7.86 9.71 load_after 8.62 8.03 9.69 +BillGates ultra-q8_0 4 b1 40.77 2087232 69a20c09ad148e2f rc0 load_before 7.59 7.83 9.60 load_after 7.63 7.77 9.50 +BillGates ultra-q8_0 5 b0 43.56 2053400 69a20c09ad148e2f rc0 load_before 6.69 7.56 9.41 load_after 8.26 7.84 9.43 +BillGates ultra-q8_0 5 b1 42.99 2088048 69a20c09ad148e2f rc0 load_before 7.15 7.62 9.34 load_after 7.83 7.71 9.28 +BillGates ultra-q8_0 5 base 43.58 2053220 69a20c09ad148e2f rc0 load_before 6.78 7.49 9.19 load_after 8.19 7.74 9.20 +BillGates redux-keep 1 base 28.11 1514576 7b7ee8115e0d94db rc0 load_before 7.24 7.55 9.12 load_after 8.16 7.75 9.14 +BillGates redux-keep 1 b0 28.55 1514820 7b7ee8115e0d94db rc0 load_before 7.81 7.69 9.11 load_after 8.47 7.84 9.11 +BillGates redux-keep 1 b1 25.00 1547740 7b7ee8115e0d94db rc0 load_before 7.48 7.64 9.03 load_after 7.84 7.71 9.02 +BillGates redux-keep 2 b0 27.74 1514676 7b7ee8115e0d94db rc0 load_before 6.94 7.52 8.94 load_after 8.15 7.76 8.98 +BillGates redux-keep 2 b1 26.02 1548692 7b7ee8115e0d94db rc0 load_before 7.20 7.57 8.90 load_after 7.90 7.69 8.91 +BillGates redux-keep 2 base 27.32 1514580 7b7ee8115e0d94db rc0 load_before 7.07 7.52 8.84 load_after 8.71 7.88 8.92 +BillGates redux-keep 3 b1 25.00 1547820 7b7ee8115e0d94db rc0 load_before 7.76 7.70 8.85 load_after 8.12 7.78 8.84 +BillGates redux-keep 3 base 27.43 1514780 7b7ee8115e0d94db rc0 load_before 7.10 7.57 8.76 load_after 7.81 7.69 8.77 +BillGates redux-keep 3 b0 27.13 1515028 7b7ee8115e0d94db rc0 load_before 7.73 7.68 8.76 load_after 8.60 7.90 8.80 +BillGates redux-keep 4 base 28.26 1514612 7b7ee8115e0d94db rc0 load_before 7.58 7.71 8.72 load_after 9.13 8.06 8.81 +BillGates redux-keep 4 b0 27.50 1514728 7b7ee8115e0d94db rc0 load_before 8.04 7.86 8.74 load_after 8.25 7.92 8.73 +BillGates redux-keep 4 b1 25.90 1546044 7b7ee8115e0d94db rc0 load_before 7.30 7.72 8.66 load_after 7.47 7.73 8.64 +BillGates redux-keep 5 b0 28.09 1514744 7b7ee8115e0d94db rc0 load_before 6.71 7.56 8.57 load_after 7.70 7.70 8.58 +BillGates redux-keep 5 b1 26.51 1548476 7b7ee8115e0d94db rc0 load_before 6.74 7.50 8.51 load_after 7.57 7.62 8.52 +BillGates redux-keep 5 base 28.25 1514496 7b7ee8115e0d94db rc0 load_before 6.71 7.44 8.45 load_after 7.11 7.46 8.43 +AimeeMullins ultra-q8_0 1 base 36.12 2005520 bc4bf944d41e6aca rc0 load_before 6.47 7.31 8.37 load_after 7.75 7.52 8.40 +AimeeMullins ultra-q8_0 1 b0 36.72 2005432 bc4bf944d41e6aca rc0 load_before 7.66 7.51 8.38 load_after 8.74 7.77 8.44 +AimeeMullins ultra-q8_0 1 b1 34.58 2040184 bc4bf944d41e6aca rc0 load_before 7.79 7.60 8.38 load_after 8.60 7.79 8.41 +AimeeMullins ultra-q8_0 2 b0 36.17 2005260 bc4bf944d41e6aca rc0 load_before 7.51 7.59 8.34 load_after 8.07 7.76 8.37 +AimeeMullins ultra-q8_0 2 b1 34.83 2040348 bc4bf944d41e6aca rc0 load_before 7.06 7.55 8.29 load_after 8.62 7.84 8.36 +AimeeMullins ultra-q8_0 2 base 36.74 2005544 bc4bf944d41e6aca rc0 load_before 7.76 7.68 8.30 load_after 8.62 7.88 8.34 +AimeeMullins ultra-q8_0 3 b1 34.47 2040380 bc4bf944d41e6aca rc0 load_before 7.69 7.70 8.28 load_after 8.55 7.92 8.33 +AimeeMullins ultra-q8_0 3 base 38.42 2005180 bc4bf944d41e6aca rc0 load_before 7.46 7.71 8.26 load_after 8.71 8.00 8.34 +AimeeMullins ultra-q8_0 3 b0 40.66 2005176 bc4bf944d41e6aca rc0 load_before 7.83 7.84 8.28 load_after 8.99 8.11 8.35 +AimeeMullins ultra-q8_0 4 base 44.01 2005284 bc4bf944d41e6aca rc0 load_before 7.91 7.91 8.28 load_after 9.98 8.43 8.44 +AimeeMullins ultra-q8_0 4 b0 36.86 2005304 bc4bf944d41e6aca rc0 load_before 8.83 8.24 8.38 load_after 9.46 8.44 8.44 +AimeeMullins ultra-q8_0 4 b1 35.10 2040732 bc4bf944d41e6aca rc0 load_before 8.39 8.24 8.38 load_after 8.98 8.41 8.43 +AimeeMullins ultra-q8_0 5 b0 37.64 2005480 bc4bf944d41e6aca rc0 load_before 7.83 8.18 8.35 load_after 8.86 8.41 8.42 +AimeeMullins ultra-q8_0 5 b1 36.13 2040572 bc4bf944d41e6aca rc0 load_before 8.11 8.26 8.37 load_after 9.07 8.48 8.44 +AimeeMullins ultra-q8_0 5 base 36.91 2005372 bc4bf944d41e6aca rc0 load_before 8.05 8.28 8.38 load_after 8.74 8.42 8.42 +AimeeMullins redux-keep 1 base 23.40 1466388 0713636b1d7ffa09 rc0 load_before 7.70 8.21 8.35 load_after 8.65 8.39 8.41 +AimeeMullins redux-keep 1 b0 23.82 1466736 0713636b1d7ffa09 rc0 load_before 7.47 8.14 8.33 load_after 8.55 8.34 8.39 +AimeeMullins redux-keep 1 b1 21.84 1502588 0713636b1d7ffa09 rc0 load_before 8.02 8.23 8.35 load_after 8.94 8.43 8.42 +AimeeMullins redux-keep 2 b0 23.67 1466420 0713636b1d7ffa09 rc0 load_before 8.18 8.28 8.37 load_after 8.00 8.23 8.35 +AimeeMullins redux-keep 2 b1 22.31 1502592 0713636b1d7ffa09 rc0 load_before 7.16 8.04 8.29 load_after 7.60 8.07 8.29 +AimeeMullins redux-keep 2 base 23.41 1466976 0713636b1d7ffa09 rc0 load_before 6.58 7.84 8.21 load_after 7.28 7.89 8.22 +AimeeMullins redux-keep 3 b1 21.89 1503932 0713636b1d7ffa09 rc0 load_before 6.54 7.71 8.16 load_after 7.19 7.76 8.16 +AimeeMullins redux-keep 3 base 23.51 1466476 0713636b1d7ffa09 rc0 load_before 6.63 7.62 8.11 load_after 7.59 7.77 8.15 +AimeeMullins redux-keep 3 b0 23.63 1466228 0713636b1d7ffa09 rc0 load_before 6.73 7.58 8.08 load_after 8.07 7.81 8.15 +AimeeMullins redux-keep 4 base 22.96 1466516 0713636b1d7ffa09 rc0 load_before 6.98 7.59 8.07 load_after 8.06 7.80 8.13 +AimeeMullins redux-keep 4 b0 22.94 1466748 0713636b1d7ffa09 rc0 load_before 7.13 7.61 8.06 load_after 8.03 7.78 8.11 +AimeeMullins redux-keep 4 b1 21.93 1502564 0713636b1d7ffa09 rc0 load_before 7.03 7.57 8.04 load_after 8.12 7.79 8.10 +AimeeMullins redux-keep 5 b0 23.79 1466556 0713636b1d7ffa09 rc0 load_before 7.26 7.61 8.04 load_after 7.93 7.73 8.06 +AimeeMullins redux-keep 5 b1 22.61 1503800 0713636b1d7ffa09 rc0 load_before 7.02 7.54 8.00 load_after 8.46 7.80 8.07 +AimeeMullins redux-keep 5 base 41.30 1466792 0713636b1d7ffa09 rc0 load_before 7.61 7.64 8.02 load_after 17.44 9.90 8.75 +JaneMcGonigal ultra-q8_0 1 base 35.54 1990188 c373acef41ee4baa rc0 load_before 4.92 7.27 8.03 load_after 7.84 7.74 8.16 +JaneMcGonigal ultra-q8_0 1 b0 35.32 1989888 c373acef41ee4baa rc0 load_before 7.01 7.56 8.10 load_after 8.66 7.91 8.20 +JaneMcGonigal ultra-q8_0 1 b1 33.31 2016792 c373acef41ee4baa rc0 load_before 7.93 7.78 8.15 load_after 8.30 7.87 8.17 +JaneMcGonigal ultra-q8_0 2 b0 34.92 1990392 c373acef41ee4baa rc0 load_before 7.41 7.70 8.11 load_after 7.92 7.77 8.12 +JaneMcGonigal ultra-q8_0 2 b1 33.95 2016712 c373acef41ee4baa rc0 load_before 6.86 7.54 8.04 load_after 7.92 7.69 8.07 +JaneMcGonigal ultra-q8_0 2 base 35.89 1989796 c373acef41ee4baa rc0 load_before 7.17 7.53 8.01 load_after 9.26 8.01 8.15 +JaneMcGonigal ultra-q8_0 3 b1 34.47 2016736 c373acef41ee4baa rc0 load_before 7.81 7.75 8.07 load_after 9.19 8.08 8.17 +JaneMcGonigal ultra-q8_0 3 base 34.97 1990160 c373acef41ee4baa rc0 load_before 8.00 7.86 8.09 load_after 8.77 8.06 8.15 +JaneMcGonigal ultra-q8_0 3 b0 34.86 1990232 c373acef41ee4baa rc0 load_before 7.65 7.84 8.08 load_after 8.46 8.01 8.12 +JaneMcGonigal ultra-q8_0 4 base 36.67 1990216 c373acef41ee4baa rc0 load_before 7.38 7.79 8.05 load_after 8.66 8.03 8.12 +JaneMcGonigal ultra-q8_0 4 b0 34.96 1989868 c373acef41ee4baa rc0 load_before 7.64 7.83 8.06 load_after 8.67 8.07 8.13 +JaneMcGonigal ultra-q8_0 4 b1 32.92 2017000 c373acef41ee4baa rc0 load_before 7.65 7.87 8.06 load_after 8.99 8.17 8.16 +JaneMcGonigal ultra-q8_0 5 b0 34.68 1989776 c373acef41ee4baa rc0 load_before 7.92 7.97 8.09 load_after 8.40 8.06 8.12 +JaneMcGonigal ultra-q8_0 5 b1 35.70 2016752 c373acef41ee4baa rc0 load_before 7.33 7.84 8.05 load_after 8.50 8.03 8.10 +JaneMcGonigal ultra-q8_0 5 base 37.53 1990160 c373acef41ee4baa rc0 load_before 7.50 7.83 8.03 load_after 9.08 8.19 8.14 +JaneMcGonigal redux-keep 1 base 24.12 1451392 3978446d6aac4afe rc0 load_before 8.06 8.00 8.08 load_after 8.62 8.13 8.12 +JaneMcGonigal redux-keep 1 b0 24.67 1451404 3978446d6aac4afe rc0 load_before 7.90 7.99 8.08 load_after 8.33 8.08 8.11 +JaneMcGonigal redux-keep 1 b1 25.57 1482340 3978446d6aac4afe rc0 load_before 7.43 7.89 8.04 load_after 8.24 8.03 8.09 +JaneMcGonigal redux-keep 2 b0 24.52 1451060 3978446d6aac4afe rc0 load_before 7.43 7.87 8.03 load_after 8.10 7.97 8.06 +JaneMcGonigal redux-keep 2 b1 39.68 1483788 3978446d6aac4afe rc0 load_before 7.48 7.83 8.02 load_after 9.59 8.33 8.17 +JaneMcGonigal redux-keep 2 base 24.96 1451660 3978446d6aac4afe rc0 load_before 8.73 8.19 8.13 load_after 9.26 8.34 8.18 +JaneMcGonigal redux-keep 3 b1 23.02 1483472 3978446d6aac4afe rc0 load_before 8.15 8.13 8.11 load_after 9.09 8.36 8.19 +JaneMcGonigal redux-keep 3 base 24.44 1450932 3978446d6aac4afe rc0 load_before 8.15 8.18 8.14 load_after 11.79 9.03 8.42 +JaneMcGonigal redux-keep 3 b0 23.15 1451448 3978446d6aac4afe rc0 load_before 10.29 8.80 8.35 load_after 10.43 8.92 8.40 +JaneMcGonigal redux-keep 4 base 22.72 1451568 3978446d6aac4afe rc0 load_before 9.05 8.67 8.32 load_after 9.94 8.92 8.42 +JaneMcGonigal redux-keep 4 b0 22.90 1451504 3978446d6aac4afe rc0 load_before 8.88 8.73 8.36 load_after 9.55 8.88 8.42 +JaneMcGonigal redux-keep 4 b1 20.00 1483192 3978446d6aac4afe rc0 load_before 8.31 8.64 8.34 load_after 8.41 8.63 8.35 +JaneMcGonigal redux-keep 5 b0 22.10 1451504 3978446d6aac4afe rc0 load_before 7.43 8.41 8.28 load_after 8.41 8.57 8.33 +JaneMcGonigal redux-keep 5 b1 21.51 1485412 3978446d6aac4afe rc0 load_before 7.51 8.37 8.27 load_after 9.80 8.81 8.42 +JaneMcGonigal redux-keep 5 base 22.87 1451368 3978446d6aac4afe rc0 load_before 9.15 8.70 8.38 load_after 11.22 9.26 8.58 +DanielKahneman ultra-q8_0 1 base 32.65 1974592 669e94614b0cba3a rc0 load_before 9.87 9.03 8.51 load_after 10.11 9.18 8.58 +DanielKahneman ultra-q8_0 1 b0 35.73 1974896 669e94614b0cba3a rc0 load_before 8.87 8.94 8.51 load_after 11.12 9.45 8.69 +DanielKahneman ultra-q8_0 1 b1 33.08 2007260 669e94614b0cba3a rc0 load_before 9.86 9.24 8.63 load_after 11.19 9.63 8.78 +DanielKahneman ultra-q8_0 2 b0 33.67 1974456 669e94614b0cba3a rc0 load_before 9.93 9.41 8.72 load_after 10.11 9.51 8.78 +DanielKahneman ultra-q8_0 2 b1 34.68 2007480 669e94614b0cba3a rc0 load_before 9.91 9.49 8.78 load_after 10.49 9.66 8.87 +DanielKahneman ultra-q8_0 2 base 35.30 1974240 669e94614b0cba3a rc0 load_before 9.35 9.44 8.80 load_after 11.10 9.87 8.97 +DanielKahneman ultra-q8_0 3 b1 32.86 2007008 669e94614b0cba3a rc0 load_before 10.18 9.71 8.93 load_after 11.09 9.98 9.05 +DanielKahneman ultra-q8_0 3 base 34.12 1974776 669e94614b0cba3a rc0 load_before 9.92 9.77 8.99 load_after 11.22 10.10 9.13 +DanielKahneman ultra-q8_0 3 b0 33.07 1974372 669e94614b0cba3a rc0 load_before 10.17 9.91 9.08 load_after 10.74 10.08 9.16 +DanielKahneman ultra-q8_0 4 base 32.30 1974504 669e94614b0cba3a rc0 load_before 9.33 9.80 9.08 load_after 10.03 9.93 9.15 +DanielKahneman ultra-q8_0 4 b0 33.05 1974652 669e94614b0cba3a rc0 load_before 9.16 9.75 9.10 load_after 9.77 9.87 9.16 +DanielKahneman ultra-q8_0 4 b1 31.43 2007432 669e94614b0cba3a rc0 load_before 8.50 9.59 9.08 load_after 9.08 9.65 9.12 +DanielKahneman ultra-q8_0 5 b0 32.67 1974588 669e94614b0cba3a rc0 load_before 8.07 9.42 9.05 load_after 9.35 9.58 9.11 +DanielKahneman ultra-q8_0 5 b1 31.29 2007608 669e94614b0cba3a rc0 load_before 8.14 9.31 9.03 load_after 9.11 9.39 9.07 +DanielKahneman ultra-q8_0 5 base 32.18 1974064 669e94614b0cba3a rc0 load_before 7.87 9.12 8.98 load_after 8.97 9.28 9.04 +DanielKahneman redux-keep 1 base 20.55 1437716 0c96f14edad70cd7 rc0 load_before 8.31 9.12 8.99 load_after 8.62 9.13 9.00 +DanielKahneman redux-keep 1 b0 21.47 1437936 0c96f14edad70cd7 rc0 load_before 7.82 8.95 8.94 load_after 9.29 9.19 9.02 +DanielKahneman redux-keep 1 b1 21.19 1470068 0c96f14edad70cd7 rc0 load_before 8.70 9.07 8.98 load_after 9.50 9.23 9.04 +DanielKahneman redux-keep 2 b0 21.65 1435688 0c96f14edad70cd7 rc0 load_before 8.66 9.06 8.98 load_after 10.83 9.51 9.13 +DanielKahneman redux-keep 2 b1 29.04 1469756 0c96f14edad70cd7 rc0 load_before 12.04 9.80 9.23 load_after 18.00 11.42 9.79 +DanielKahneman redux-keep 2 base 23.07 1434520 0c96f14edad70cd7 rc0 load_before 18.00 11.64 9.87 load_after 17.59 12.10 10.08 +DanielKahneman redux-keep 3 b1 20.79 1469588 0c96f14edad70cd7 rc0 load_before 16.32 12.01 10.07 load_after 14.59 11.89 10.07 +DanielKahneman redux-keep 3 base 21.41 1435020 0c96f14edad70cd7 rc0 load_before 12.64 11.56 9.99 load_after 11.88 11.46 9.99 +DanielKahneman redux-keep 3 b0 24.15 1438240 0c96f14edad70cd7 rc0 load_before 10.66 11.21 9.92 load_after 12.34 11.58 10.08 +DanielKahneman redux-keep 4 base 20.22 1434756 0c96f14edad70cd7 rc0 load_before 10.91 11.30 10.00 load_after 10.23 11.10 9.97 +DanielKahneman redux-keep 4 b0 20.04 1437892 0c96f14edad70cd7 rc0 load_before 9.12 10.83 9.89 load_after 9.00 10.68 9.86 +DanielKahneman redux-keep 4 b1 19.09 1469668 0c96f14edad70cd7 rc0 load_before 7.92 10.40 9.78 load_after 9.82 10.65 9.87 +DanielKahneman redux-keep 5 b0 21.08 1437192 0c96f14edad70cd7 rc0 load_before 8.78 10.40 9.80 load_after 9.08 10.35 9.79 +DanielKahneman redux-keep 5 b1 24.75 1469552 0c96f14edad70cd7 rc0 load_before 7.92 10.05 9.70 load_after 9.67 10.31 9.80 +DanielKahneman redux-keep 5 base 23.87 1434984 0c96f14edad70cd7 rc0 load_before 8.72 10.08 9.73 load_after 11.06 10.50 9.88 diff --git a/scripts/vad_bench/results/silero_parity_results.txt b/scripts/vad_bench/results/silero_parity_results.txt new file mode 100644 index 0000000..8d4ab81 --- /dev/null +++ b/scripts/vad_bench/results/silero_parity_results.txt @@ -0,0 +1,50 @@ +PROBABILITY PARITY vs onnxruntime (v6.2.3 ONNX, official 64-sample context), all 120 clips + 600 s talk, per 32 ms chunk +system | max abs diff | mean abs diff | frames > 0.05 off | decision agreement @0.5 | flipped frames +wcpp_conv623 | 0.7373 | 0.00224 | 0.54% | 99.826% | 309 +wcpp_repo620 | 0.7373 | 0.00224 | 0.54% | 99.826% | 309 +pk_f32 | 0.0001 | 0.00002 | 0.00% | 99.999% | 2 +pk_f16 | 0.0036 | 0.00005 | 0.00% | 99.997% | 5 +pairwise (whisper.cpp converted v6.2.3 vs parakeet f32): +wcpp_conv623 vs pk_f32 | 0.7373 | 0.00224 | 0.54% | 99.825% | 311 +wcpp_conv623 vs wcpp_repo620 | 0.0000 | 0.00000 | 0.00% | 100.000% | 0 +per-region parity vs onnx (mean abs diff) : first chunk of each clip / rest +wcpp_conv623 first-chunk 0.00088 rest 0.00212 +wcpp_repo620 first-chunk 0.00088 rest 0.00212 +pk_f32 first-chunk 0.00002 rest 0.00002 +pk_f16 first-chunk 0.00005 rest 0.00005 + +SEGMENT QUALITY, frame level on 10 ms grid vs synthetic reference (P/R/F1 %), 40 clips per condition +matched post = thr 0.5 / neg 0.35 / min speech 100 ms / min silence 200 ms / pad 0 (same python code on every probability stream) +system | clean | white10 | pink10 | all +onnx [matched post] | 99.3/89.3/94.0 | 99.4/87.8/93.2 | 99.5/87.4/93.1 | 99.4/88.2/93.5 +wcpp_conv623 [matched post] | 99.3/89.4/94.1 | 99.4/87.8/93.3 | 99.5/87.4/93.0 | 99.4/88.2/93.5 +pk_f32 [matched post] | 99.3/89.3/94.0 | 99.4/87.8/93.2 | 99.5/87.4/93.1 | 99.4/88.2/93.5 +pk_f16 [matched post] | 99.3/89.3/94.0 | 99.4/87.8/93.2 | 99.5/87.4/93.1 | 99.4/88.2/93.5 +wcpp_conv623 [whisper.cpp own default] | 98.9/90.3/94.4 | 99.2/89.0/93.8 | 99.3/88.6/93.6 | 99.1/89.3/94.0 +wcpp_conv623 [whisper.cpp C++ post, matched params] | 99.3/89.4/94.1 | 99.4/87.8/93.3 | 99.5/87.4/93.0 | 99.4/88.2/93.5 +pk_f32 [parakeet own default] | 99.1/89.3/93.9 | 99.3/87.6/93.1 | 99.4/87.3/93.0 | 99.3/88.1/93.3 + +BOUNDARY ERROR vs reference, all conditions pooled: start median/p90, end median/p90, miss% +onnx [matched post] | start 241/553 | end 108/396 | miss 0.2% +wcpp_conv623 [matched post] | start 241/553 | end 108/396 | miss 0.2% +pk_f32 [matched post] | start 241/553 | end 108/396 | miss 0.2% +pk_f16 [matched post] | start 241/553 | end 108/396 | miss 0.2% +wcpp_conv623 [whisper.cpp own default] | start 209/521 | end 124/367 | miss 0.0% +wcpp_conv623 [whisper.cpp C++ post, matched params] | start 237/551 | end 110/398 | miss 0.2% +pk_f32 [parakeet own default] | start 211/523 | end 116/392 | miss 0.0% + +AGREEMENT of speech masks (matched post), all 120 clips pooled: F1 / kappa / IoU +wcpp_conv623 vs pk_f32 | F1 99.92 | kappa 0.9969 | IoU 99.84 +wcpp_conv623 vs onnx | F1 99.92 | kappa 0.9969 | IoU 99.84 +pk_f32 vs onnx | F1 100.00 | kappa 1.0000 | IoU 100.00 +pk_f16 vs pk_f32 | F1 100.00 | kappa 0.9999 | IoU 100.00 +wcpp_conv623 vs wcpp_repo620 | F1 100.00 | kappa 1.0000 | IoU 100.00 + +AGREEMENT on the 600 s TED talk (no reference): F1 / kappa, speech fraction +onnx speech fraction 79.1% +wcpp_conv623 speech fraction 78.8% +pk_f32 speech fraction 79.1% +wcpp_conv623 vs pk_f32 | F1 99.75 | kappa 0.9882 +wcpp_conv623 vs onnx | F1 99.75 | kappa 0.9882 +pk_f32 vs onnx | F1 100.00 | kappa 1.0000 +TED talk flipped frames @0.5 vs onnx: wcpp_conv623 57/18750, wcpp_repo620 57/18750, pk_f32 0/18750, pk_f16 1/18750 \ No newline at end of file diff --git a/scripts/vad_bench/results/silero_speed2_raw.json b/scripts/vad_bench/results/silero_speed2_raw.json new file mode 100644 index 0000000..a3cc405 --- /dev/null +++ b/scripts/vad_bench/results/silero_speed2_raw.json @@ -0,0 +1,110 @@ +[ + { + "job": "onnxruntime_python_1thr_detect_only", + "wall": 0.8058516788296402, + "inproc": 0.6678, + "rss_kb": 106744, + "user": 0.76, + "mean_runnable": 4.8, + "load1": 4.9, + "quiet": true, + "attempts": 1, + "round": 0 + }, + { + "job": "whisper-cli_tiny.en_novad", + "wall": 15.08737738407217, + "inproc": null, + "rss_kb": 296592, + "user": 14.98, + "mean_runnable": 4.0, + "load1": 4.69, + "quiet": true, + "attempts": 1, + "round": 0 + }, + { + "job": "whisper-cli_tiny.en_vad", + "wall": 19.682566916104406, + "inproc": null, + "rss_kb": 314876, + "user": 18.18, + "mean_runnable": 4.4, + "load1": 4.59, + "quiet": true, + "attempts": 1, + "round": 0 + }, + { + "job": "whisper-cli_tiny.en_novad", + "wall": 15.358130356995389, + "inproc": null, + "rss_kb": 296748, + "user": 15.26, + "mean_runnable": 3.7, + "load1": 6.22, + "quiet": true, + "attempts": 2, + "round": 1 + }, + { + "job": "whisper-cli_tiny.en_vad", + "wall": 19.611542615108192, + "inproc": null, + "rss_kb": 314880, + "user": 18.22, + "mean_runnable": 3.7, + "load1": 7.3, + "quiet": true, + "attempts": 11, + "round": 1 + }, + { + "job": "onnxruntime_python_1thr_detect_only", + "wall": 1.241450318833813, + "inproc": 0.7934, + "rss_kb": 106540, + "user": 0.69, + "mean_runnable": 3.0, + "load1": 6.26, + "quiet": true, + "attempts": 1, + "round": 1 + }, + { + "job": "whisper-cli_tiny.en_vad", + "wall": 20.62023510481231, + "inproc": null, + "rss_kb": 314540, + "user": 18.96, + "mean_runnable": 5.4, + "load1": 4.6, + "quiet": true, + "attempts": 10, + "round": 2 + }, + { + "job": "onnxruntime_python_1thr_detect_only", + "wall": 0.7381441139150411, + "inproc": 0.607, + "rss_kb": 106712, + "user": 0.71, + "mean_runnable": 2.8, + "load1": 6.91, + "quiet": true, + "attempts": 4, + "round": 2 + }, + { + "job": "whisper-cli_tiny.en_novad", + "wall": 15.142986833816394, + "inproc": null, + "rss_kb": 296756, + "user": 15.04, + "mean_runnable": 2.9, + "load1": 8.26, + "quiet": true, + "attempts": 11, + "round": 2 + } +] \ No newline at end of file diff --git a/scripts/vad_bench/results/silero_speed_raw.json b/scripts/vad_bench/results/silero_speed_raw.json new file mode 100644 index 0000000..229868c --- /dev/null +++ b/scripts/vad_bench/results/silero_speed_raw.json @@ -0,0 +1,242 @@ +[ + { + "job": "wcpp_inproc_detect_only", + "wall": 0.44647544994950294, + "inproc": 0.4318, + "rss_kb": 36960, + "user": 0.43, + "mean_runnable": 2.5, + "load1": 14.27, + "quiet": true, + "attempts": 1, + "round": 0 + }, + { + "job": "wcpp_vad-speech-segments_cli", + "wall": 0.4505736359860748, + "inproc": null, + "rss_kb": 28800, + "user": 0.44, + "mean_runnable": 3.2, + "load1": 14.01, + "quiet": true, + "attempts": 1, + "round": 0 + }, + { + "job": "parakeet_cli_f32", + "wall": 0.4249067639466375, + "inproc": null, + "rss_kb": 61120, + "user": 0.4, + "mean_runnable": 1.8, + "load1": 13.79, + "quiet": true, + "attempts": 1, + "round": 0 + }, + { + "job": "parakeet_cli_f16", + "wall": 0.436342905042693, + "inproc": null, + "rss_kb": 61120, + "user": 0.41, + "mean_runnable": 2.9, + "load1": 13.44, + "quiet": true, + "attempts": 1, + "round": 0 + }, + { + "job": "wcpp_vad-speech-segments_cli", + "wall": 0.45197289693169296, + "inproc": null, + "rss_kb": 28640, + "user": 0.44, + "mean_runnable": 2.3, + "load1": 13.36, + "quiet": true, + "attempts": 1, + "round": 1 + }, + { + "job": "parakeet_cli_f32", + "wall": 0.4639807508792728, + "inproc": null, + "rss_kb": 61120, + "user": 0.44, + "mean_runnable": 2.7, + "load1": 13.29, + "quiet": true, + "attempts": 1, + "round": 1 + }, + { + "job": "parakeet_cli_f16", + "wall": 0.47317962581291795, + "inproc": null, + "rss_kb": 61120, + "user": 0.44, + "mean_runnable": 3.8, + "load1": 14.99, + "quiet": true, + "attempts": 2, + "round": 1 + }, + { + "job": "wcpp_inproc_detect_only", + "wall": 0.45590541488491, + "inproc": 0.4372, + "rss_kb": 36960, + "user": 0.44, + "mean_runnable": 3.2, + "load1": 15.93, + "quiet": true, + "attempts": 1, + "round": 1 + }, + { + "job": "parakeet_cli_f32", + "wall": 0.4558734449092299, + "inproc": null, + "rss_kb": 61120, + "user": 0.43, + "mean_runnable": 3.5, + "load1": 15.38, + "quiet": true, + "attempts": 2, + "round": 2 + }, + { + "job": "parakeet_cli_f16", + "wall": 0.5525356309954077, + "inproc": null, + "rss_kb": 61120, + "user": 0.47, + "mean_runnable": 3.8, + "load1": 13.82, + "quiet": true, + "attempts": 9, + "round": 2 + }, + { + "job": "wcpp_inproc_detect_only", + "wall": 0.46314642787911, + "inproc": 0.4413, + "rss_kb": 36800, + "user": 0.44, + "mean_runnable": 3.5, + "load1": 7.94, + "quiet": true, + "attempts": 5, + "round": 2 + }, + { + "job": "wcpp_vad-speech-segments_cli", + "wall": 0.45731914788484573, + "inproc": null, + "rss_kb": 28640, + "user": 0.44, + "mean_runnable": 2.6, + "load1": 7.26, + "quiet": true, + "attempts": 1, + "round": 2 + }, + { + "job": "parakeet_cli_f16", + "wall": 0.4371650768443942, + "inproc": null, + "rss_kb": 61120, + "user": 0.41, + "mean_runnable": 3.6, + "load1": 7.19, + "quiet": true, + "attempts": 4, + "round": 3 + }, + { + "job": "wcpp_inproc_detect_only", + "wall": 0.4506124691106379, + "inproc": 0.4376, + "rss_kb": 36960, + "user": 0.43, + "mean_runnable": 3.0, + "load1": 4.75, + "quiet": true, + "attempts": 3, + "round": 3 + }, + { + "job": "wcpp_vad-speech-segments_cli", + "wall": 0.4435456038918346, + "inproc": null, + "rss_kb": 28480, + "user": 0.43, + "mean_runnable": 2.6, + "load1": 4.4, + "quiet": true, + "attempts": 1, + "round": 3 + }, + { + "job": "parakeet_cli_f32", + "wall": 0.45499568711966276, + "inproc": null, + "rss_kb": 61120, + "user": 0.42, + "mean_runnable": 4.9, + "load1": 4.51, + "quiet": true, + "attempts": 1, + "round": 3 + }, + { + "job": "wcpp_inproc_detect_only", + "wall": 0.47262479993514717, + "inproc": 0.4548, + "rss_kb": 36800, + "user": 0.45, + "mean_runnable": 3.3, + "load1": 4.35, + "quiet": true, + "attempts": 1, + "round": 4 + }, + { + "job": "wcpp_vad-speech-segments_cli", + "wall": 0.45728860003873706, + "inproc": null, + "rss_kb": 28640, + "user": 0.44, + "mean_runnable": 4.2, + "load1": 4.29, + "quiet": true, + "attempts": 1, + "round": 4 + }, + { + "job": "parakeet_cli_f32", + "wall": 0.4525157401803881, + "inproc": null, + "rss_kb": 61120, + "user": 0.42, + "mean_runnable": 5.5, + "load1": 4.46, + "quiet": true, + "attempts": 1, + "round": 4 + }, + { + "job": "parakeet_cli_f16", + "wall": 0.45077538792975247, + "inproc": null, + "rss_kb": 61120, + "user": 0.42, + "mean_runnable": 4.1, + "load1": 4.16, + "quiet": true, + "attempts": 1, + "round": 4 + } +] \ No newline at end of file diff --git a/scripts/vad_bench/results/silero_versions_and_sha256.txt b/scripts/vad_bench/results/silero_versions_and_sha256.txt new file mode 100644 index 0000000..4156c5a --- /dev/null +++ b/scripts/vad_bench/results/silero_versions_and_sha256.txt @@ -0,0 +1,16 @@ +whisper.cpp 60c0be6ac8fa71b1a2ae2dd938a31a34a508e774 Fri Oct 2 15:21:13 2026 +0200 +set(WHISPER_VERSION_MAJOR 1) +parakeet.cpp c1a8391102d34074f48e758c441758327d2465b7 Merge origin/master into feat/vad-silero-api +parakeet ggml e705c5fe +gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 +cmake version 3.28.3 +8160489282352accc0e95925c2f6bf3d76fb8f7bccdce5c7e46808cb15e8443c silero-vad-f16.gguf +1398e5ce230bd8f20c06f825ed3a41dc4528ef7e6fe8f8ec8a5b7aedefbee943 silero-vad-f32.gguf +1a153a22f4509e292a94e67d6f9b85e8deb25b4988682b7e174c65279d8788e3 silero_vad.onnx +2aa269b785eeb53a82983a20501ddf7c1d9c48e33ab63a41391ac6c9f7fb6987 ggml-silero-v6.2.0.bin +abd79739d3ff4bc37fa1d24f1fca8244f2e31ec9f0bb7ea6244b185bfaa0b8e6 ggml-silero-v6.2.3-ggml.bin +921e4cf8686fdd993dcd081a5da5b6c365bfde1162e72b08d75ac75289920b1f ggml-tiny.en.bin +5b8e79aac593eb87aab62e2083d63e557570d7376b73e8f50c08adfbdbb69767 whisper.cpp/build/bin/whisper-cli +bf89b74f4a104ff32996eca405b5d5a13fefcdfb74882e5f131b849be37276ea whisper.cpp/build/bin/whisper-vad-speech-segments +31bc6b76f9929042e9b47bac13f521c732dc072818df81bee01531fcfc558aec wvad +ed5bb79c419fef6e8e31a3fee650fbdac2ed3c931f03102bbd6f5b7c04a0879e build/examples/cli/parakeet-cli diff --git a/scripts/vad_bench/silero_analyze.py b/scripts/vad_bench/silero_analyze.py new file mode 100755 index 0000000..414da92 --- /dev/null +++ b/scripts/vad_bench/silero_analyze.py @@ -0,0 +1,93 @@ +import json,os,sys,numpy as np +from vad_synth import * +SB=sys.argv[1] # DATA_DIR, as for silero_collect.py; run in the directory that holds probs.npz +P=np.load("probs.npz"); S=json.load(open("segs_own_default.json")) +meta=json.load(open(SB+"/clips.json")) +SYS=["onnx","wcpp_conv623","wcpp_repo620","pk_f32","pk_f16"] +conds=["clean0","clean","white10","pink10"] +def post(p,thr=0.5,min_speech=0.1,min_sil=0.2,pad=0.0,fs=0.032,total=None): + # silero get_speech_timestamps-style hysteresis (neg = thr-0.15), no max-speech split + neg=max(thr-0.15,0.01); segs=[];cur=None;tmp=None + for i,v in enumerate(p): + t=i*fs + if v>=thr and tmp is not None: tmp=None + if v>=thr and cur is None: cur=t; continue + if vmin_speech: segs.append([cur,tmp]) + cur=None;tmp=None + end=len(p)*fs + if cur is not None and end-cur>min_speech: segs.append([cur,end]) + out=[] + for i,(a,b) in enumerate(segs): + out.append([max(0,a-pad),min(total or end,b+pad)]) + return out +def lens(a,b): n=min(len(a),len(b)); return a[:n],b[:n] +# ---- 1. probability parity (vs onnxruntime) ---- +names=[m["name"] for m in meta] +def parity(sysn,ref="onnx",nms=names+["ted_talk"]): + mx=0;sm=0;cnt=0;dec=0;d1=0;big=0 + for n in nms: + a,b=lens(P[f"{n}|{sysn}"],P[f"{n}|{ref}"]); d=np.abs(a-b) + mx=max(mx,d.max());sm+=d.sum();cnt+=len(d);dec+=((a>=.5)!=(b>=.5)).sum();big+=(d>0.05).sum() + return mx,sm/cnt,100*(1-dec/cnt),dec,100*big/cnt +out=[] +out.append("PROBABILITY PARITY vs onnxruntime (v6.2.3 ONNX, official 64-sample context), all 120 clips + 600 s talk, per 32 ms chunk") +out.append("system | max abs diff | mean abs diff | frames > 0.05 off | decision agreement @0.5 | flipped frames") +for s in SYS[1:]: + mx,mn,ag,fl,big=parity(s); out.append(f"{s} | {mx:.4f} | {mn:.5f} | {big:.2f}% | {ag:.3f}% | {fl}") +out.append("pairwise (whisper.cpp converted v6.2.3 vs parakeet f32):") +mx,mn,ag,fl,big=parity("wcpp_conv623","pk_f32");out.append(f"wcpp_conv623 vs pk_f32 | {mx:.4f} | {mn:.5f} | {big:.2f}% | {ag:.3f}% | {fl}") +mx,mn,ag,fl,big=parity("wcpp_conv623","wcpp_repo620");out.append(f"wcpp_conv623 vs wcpp_repo620 | {mx:.4f} | {mn:.5f} | {big:.2f}% | {ag:.3f}% | {fl}") +out.append("per-region parity vs onnx (mean abs diff) : first chunk of each clip / rest") +for s in SYS[1:]: + f=np.mean([abs(P[f"{n}|{s}"][0]-P[f"{n}|onnx"][0]) for n in names]); r=np.mean([np.abs(*[lens(P[f"{n}|{s}"],P[f"{n}|onnx"])[0]-lens(P[f"{n}|{s}"],P[f"{n}|onnx"])[1]])[1:].mean() for n in names]) + out.append(f"{s} first-chunk {f:.5f} rest {r:.5f}") +# ---- 2. segment quality ---- +def agg(getseg,cs): + c=np.zeros(4);bs=[];be=[];miss=0;nb=0 + for m in meta: + if m["cond"] not in cs: continue + L=m["dur"]; pr=getseg(m["name"],L); ref=m["ref"] + c+=counts(mask(pr,L),mask(ref,L)) + e1,m1=bound_err(pr,ref,0);e2,m2=bound_err(pr,ref,1);bs+=e1;be+=e2;miss+=m1+m2;nb+=2*len(ref) + return c,bs,be,miss,nb +sysdefs={} +for s in ["onnx","wcpp_conv623","pk_f32","pk_f16"]: + sysdefs[f"{s} [matched post]"]=(lambda s:lambda n,L:post(P[f"{n}|{s}"],total=L))(s) +sysdefs["wcpp_conv623 [whisper.cpp own default]"]=lambda n,L:S[n]["wcpp_conv623_seg_default"] +sysdefs["wcpp_conv623 [whisper.cpp C++ post, matched params]"]=lambda n,L:S[n]["wcpp_conv623_seg_matched"] +sysdefs["pk_f32 [parakeet own default]"]=lambda n,L:S[n]["pk_f32_seg_default"] +out.append("\nSEGMENT QUALITY, frame level on 10 ms grid vs synthetic reference (P/R/F1 %), 40 clips per condition") +out.append("matched post = thr 0.5 / neg 0.35 / min speech 100 ms / min silence 200 ms / pad 0 (same python code on every probability stream)") +allc=["clean","white10","pink10"] +out.append("system | "+" | ".join(allc)+" | all") +res={} +for k,g in sysdefs.items(): + cells=[] + for cs in [[c] for c in allc]+[allc]: + c,*_=agg(g,cs); p,r,f=prf(c); cells.append(f"{p*100:.1f}/{r*100:.1f}/{f*100:.1f}") + out.append(f"{k} | "+" | ".join(cells)) +out.append("\nBOUNDARY ERROR vs reference, all conditions pooled: start median/p90, end median/p90, miss%") +for k,g in sysdefs.items(): + c,bs,be,miss,nb=agg(g,allc) + out.append(f"{k} | start {np.median(bs):.0f}/{np.percentile(bs,90):.0f} | end {np.median(be):.0f}/{np.percentile(be,90):.0f} | miss {100*miss/nb:.1f}%") +# ---- 4. agreement between implementations ---- +out.append("\nAGREEMENT of speech masks (matched post), all 120 clips pooled: F1 / kappa / IoU") +pairs=[("wcpp_conv623","pk_f32"),("wcpp_conv623","onnx"),("pk_f32","onnx"),("pk_f16","pk_f32"),("wcpp_conv623","wcpp_repo620")] +def agree(a,b,nms,Ls): + c=np.zeros(4) + for n,L in zip(nms,Ls): c+=counts(mask(post(P[f"{n}|{a}"],total=L),L),mask(post(P[f"{n}|{b}"],total=L),L)) + return c +for a,b in pairs: + c=agree(a,b,names,[m["dur"] for m in meta]); p,r,f=prf(c) + out.append(f"{a} vs {b} | F1 {f*100:.2f} | kappa {kappa(c):.4f} | IoU {100*c[0]/(c[0]+c[1]+c[2]):.2f}") +out.append("\nAGREEMENT on the 600 s TED talk (no reference): F1 / kappa, speech fraction") +L=600.0 +for s in ["onnx","wcpp_conv623","pk_f32"]: + m=mask(post(P[f"ted_talk|{s}"],total=L),L); out.append(f"{s} speech fraction {100*m.mean():.1f}%") +for a,b in pairs[:3]: + c=agree(a,b,["ted_talk"],[L]); p,r,f=prf(c); out.append(f"{a} vs {b} | F1 {f*100:.2f} | kappa {kappa(c):.4f}") +out.append("TED talk flipped frames @0.5 vs onnx: "+", ".join(f"{s} {int(((lens(P[f'ted_talk|{s}'],P['ted_talk|onnx'])[0]>=.5)!=(lens(P[f'ted_talk|{s}'],P['ted_talk|onnx'])[1]>=.5)).sum())}/{len(P['ted_talk|onnx'])}" for s in SYS[1:])) +open("results.txt","w").write("\n".join(out)); print("\n".join(out)) diff --git a/scripts/vad_bench/silero_collect.py b/scripts/vad_bench/silero_collect.py new file mode 100755 index 0000000..b335ffe --- /dev/null +++ b/scripts/vad_bench/silero_collect.py @@ -0,0 +1,41 @@ +import json,os,subprocess,sys,numpy as np,soundfile as sf,onnxruntime as ort +# usage: collect.py CORE DATA_DIR MODEL_DIR PARAKEET_CLI WVAD +# DATA_DIR: clips/ clips.json ted_talk.wav (make_clips.py, fetch_data.py) +# MODEL_DIR: silero_vad.onnx silero-vad-f32.gguf silero-vad-f16.gguf ggml-silero-v6.2.3-ggml.bin ggml-silero-v6.2.0.bin +CORE,SB,M,PK,WV=sys.argv[1:6] +TS=["taskset","-c",CORE] +clips=json.load(open(SB+"/clips.json")) +files=[f"{SB}/clips/{c['name']}.wav" for c in clips]+[SB+"/ted_talk.wav"] +names=[c["name"] for c in clips]+["ted_talk"] +so=ort.SessionOptions(); so.intra_op_num_threads=1; so.inter_op_num_threads=1 +sess=ort.InferenceSession(M+"/silero_vad.onnx",so,providers=["CPUExecutionProvider"]) +def onnx_probs(y): + n=(len(y)+511)//512; y=np.pad(y,(0,n*512-len(y))) + st=np.zeros((2,1,128),np.float32); ctx=np.zeros(64,np.float32); out=[] + for i in range(n): + x=np.concatenate([ctx,y[i*512:(i+1)*512]])[None].astype(np.float32) + o,st=sess.run(None,{"input":x,"state":st,"sr":np.array(16000,np.int64)}) + out.append(float(o[0,0])); ctx=x[0,-64:] + return np.array(out) +res={} +def wh(model,tag): + p=subprocess.run(TS+[WV,"probs",model,"1"]+files,capture_output=True,text=True,check=True).stdout.strip().split("\n") + for nm,l in zip(names,p): + j=json.loads(l); res.setdefault(nm,{})[tag]=np.array(j["probs"]); res[nm][tag+"_seg_default"]=j["default"]; res[nm][tag+"_seg_matched"]=j["matched"] +def pk(model,tag): + for nm,f in zip(names,files): + j=json.loads(subprocess.run(TS+[PK,"vad","--model",model,"--input",f,"--probabilities","--threads","1"],capture_output=True,text=True,check=True).stdout) + res.setdefault(nm,{})[tag]=np.array(j["probabilities"]); res[nm][tag+"_seg_default"]=[[s["start"],s["end"]] for s in j["segments"]] +wh(M+"/ggml-silero-v6.2.3-ggml.bin","wcpp_conv623"); print("wcpp conv done",flush=True) +wh(M+"/ggml-silero-v6.2.0.bin","wcpp_repo620"); print("wcpp repo done",flush=True) +pk(M+"/silero-vad-f32.gguf","pk_f32"); print("pk f32",flush=True) +pk(M+"/silero-vad-f16.gguf","pk_f16"); print("pk f16",flush=True) +for nm,f in zip(names,files): + y=sf.read(f,dtype="float32")[0]; res[nm]["onnx"]=onnx_probs(y) +print("onnx done",flush=True) +flat={} +for nm,d in res.items(): + for k,v in d.items(): + if isinstance(v,np.ndarray): flat[f"{nm}|{k}"]=v +np.savez_compressed("probs.npz",**flat) +json.dump({nm:{k:v for k,v in d.items() if not isinstance(v,np.ndarray)} for nm,d in res.items()},open("segs_own_default.json","w")) diff --git a/scripts/vad_bench/speed_heads.py b/scripts/vad_bench/speed_heads.py new file mode 100755 index 0000000..0caffb6 --- /dev/null +++ b/scripts/vad_bench/speed_heads.py @@ -0,0 +1,18 @@ +#!/usr/bin/env python3 +"""usage: PARAKEET_LIB=... taskset -c CPUS speed_heads.py TALK.wav GGUF_DIR +VAD speed on the first 300 s: Silero ONNX on 1 thread vs the heads (8 threads), 4 reps.""" +import sys,time,subprocess,numpy as np,soundfile as sf,torch +from heads_lib import silero,parakeet +from pk import PK +y=sf.read(sys.argv[1],dtype="float32")[0][:16000*300] # 300 s +D=len(y)/16000 +pks={"ultra_q8":PK(sys.argv[2]+"/ultra-q8_0.gguf"),"ultra_f16":PK(sys.argv[2]+"/ultra-f16.gguf"),"redux_packed":PK(sys.argv[2]+"/redux-keep.gguf"),"redux_deq_f16":PK(sys.argv[2]+"/redux-deq-f16.gguf")} +fns={"silero_onnx_1thr":lambda:silero(y,True)} +for n,p in pks.items(): fns[n+"_8thr"]=(lambda p=p:parakeet(p,y)) +T={k:[] for k in fns}; loads=[] +for rep in range(4): + loads.append(subprocess.getoutput("cut -d' ' -f1 /proc/loadavg")) + for k,f in fns.items(): + t0=time.perf_counter();f();T[k].append(time.perf_counter()-t0) +print("load1 at each rep start:",loads) +for k,v in T.items(): print("%-18s best %.2fs median %.2fs -> %.0fx / %.0fx real time"%(k,min(v),np.median(v),D/min(v),D/np.median(v))) diff --git a/scripts/vad_bench/speed_silero.py b/scripts/vad_bench/speed_silero.py new file mode 100755 index 0000000..98c83f3 --- /dev/null +++ b/scripts/vad_bench/speed_silero.py @@ -0,0 +1,71 @@ +#!/usr/bin/env python3 +"""Timing of Silero VAD on a 300 s clip, one pinned core, serialized with a lock. + +usage: speed_silero.py CORE ROUNDS WAV300 MODEL_DIR PARAKEET_CLI WVAD WHISPER_BIN_DIR OUT.json [--lock FILE] + +Jobs (round-robin order, one run each per round): + wcpp_inproc_detect_only wvad time: whisper_vad_detect_speech only, model already loaded + wcpp_vad-speech-segments_cli whisper-vad-speech-segments (model load + detect + print) + parakeet_cli_f32 / _f16 parakeet-cli vad (model load + detect + print) + onnxruntime_python_1thr_detect_only onnx_time.py (Python loop, one 512-sample call per chunk) + whisper-cli_tiny.en_novad / _vad whisper-cli end to end with tiny.en, 1 thread (needs ggml-tiny.en.bin) +Before each run the script waits until the mean number of other runnable tasks over 10 s is below +4 or the 1-minute load average is below 5 (it gives up waiting after 30 tries and runs anyway, and +records quiet=false). It reports wall time, user time, max RSS (from /usr/bin/time -v), the mean +runnable count and the load average at the start. MODEL_DIR: see silero_collect.py. +""" +import fcntl, json, os, re, subprocess, sys, tempfile, time +import numpy as np + +CORE, ROUNDS, WAV, M, PK, WV, WB, OUT = sys.argv[1], int(sys.argv[2]), *sys.argv[3:9] +LOCK = sys.argv[sys.argv.index("--lock") + 1] if "--lock" in sys.argv else "/tmp/pk-bench.lock" +WS, WC = WB + "/whisper-vad-speech-segments", WB + "/whisper-cli" +VM = M + "/ggml-silero-v6.2.3-ggml.bin"; TM = M + "/ggml-tiny.en.bin" +HERE = os.path.dirname(os.path.abspath(__file__)) +JOBS = { + "wcpp_inproc_detect_only": [WV, "time", VM, "1", "1", WAV], + "wcpp_vad-speech-segments_cli": [WS, "-vm", VM, "-f", WAV, "-t", "1", "-np"], + "parakeet_cli_f32": [PK, "vad", "--model", M + "/silero-vad-f32.gguf", "--input", WAV, "--threads", "1"], + "parakeet_cli_f16": [PK, "vad", "--model", M + "/silero-vad-f16.gguf", "--input", WAV, "--threads", "1"], + "onnxruntime_python_1thr_detect_only": ["python3", HERE + "/onnx_time.py", M + "/silero_vad.onnx", WAV], +} +if os.path.exists(TM): + JOBS["whisper-cli_tiny.en_novad"] = [WC, "-m", TM, "-f", WAV, "-t", "1", "-np", "-nt"] + JOBS["whisper-cli_tiny.en_vad"] = [WC, "-m", TM, "-f", WAV, "-t", "1", "-np", "-nt", "--vad", "-vm", VM] +INPROC = ("wcpp_inproc", "onnxruntime") + +def runnable(): + return int(open("/proc/loadavg").read().split()[3].split("/")[0]) - 1 # minus this process + +def gate(): + s = [] + for _ in range(10): + s.append(runnable()); time.sleep(1) + return float(np.mean(s)), float(open("/proc/loadavg").read().split()[0]) + +lockf = open(LOCK, "a") +tf = tempfile.NamedTemporaryFile(suffix=".time", delete=False).name + +def timed(name): + cmd = ["taskset", "-c", CORE, "/usr/bin/time", "-v", "-o", tf] + JOBS[name] + for attempt in range(30): + fcntl.flock(lockf, fcntl.LOCK_EX) + try: + mr, l1 = gate(); quiet = (mr < 4 or l1 < 5) + if quiet or attempt == 29: + t0 = time.perf_counter(); p = subprocess.run(cmd, capture_output=True, text=True); wall = time.perf_counter() - t0 + tt = open(tf).read() + rss = int(re.search(r"Maximum resident set size \(kbytes\): (\d+)", tt).group(1)) + user = float(re.search(r"User time \(seconds\): ([\d.]+)", tt).group(1)) + inproc = float(p.stdout.strip().split()[0]) if name.startswith(INPROC) else None + return dict(job=name, wall=wall, inproc=inproc, rss_kb=rss, user=user, mean_runnable=mr, load1=l1, quiet=quiet, attempts=attempt + 1) + finally: + fcntl.flock(lockf, fcntl.LOCK_UN) + time.sleep(20) + +names = list(JOBS); rows = [] +for r in range(ROUNDS): + for n in names[r % len(names):] + names[:r % len(names)]: + x = timed(n); x["round"] = r; rows.append(x); print(json.dumps(x), flush=True) +os.unlink(tf) +json.dump(rows, open(OUT, "w"), indent=1) diff --git a/scripts/vad_bench/ted_reference.py b/scripts/vad_bench/ted_reference.py new file mode 100755 index 0000000..a0ecbad --- /dev/null +++ b/scripts/vad_bench/ted_reference.py @@ -0,0 +1,19 @@ +#!/usr/bin/env python3 +"""usage: ted_reference.py DATA_DIR TDT_0.6B_V3_F16.gguf +Word timestamps of each talk from the ASR model (30 s clips, batches of 8); compare_ted.py merges +words closer than 0.4 s into speech spans. The reference is therefore coarse and ASR-derived.""" +import json,glob,sys,soundfile as sf,numpy as np +from pk import PK +pk=PK(sys.argv[2]) +for f in sorted(glob.glob(sys.argv[1]+"/ted_*.wav")): + y,_=sf.read(f,dtype="float32") + if len(y)<16000*60: continue + W=[];C=30*16000 + clips=[y[i:i+C] for i in range(0,len(y),C)] + for b in range(0,len(clips),8): + res=pk.words_batch(clips[b:b+8]) + for k,r in enumerate(res): + off=(b+k)*30.0 + for w in r["words"]: W.append([w["start"]+off,w["end"]+off]) + json.dump(W,open(f.replace(".wav","_words.json"),"w")) + print(f,len(y)/16000,len(W),flush=True) diff --git a/scripts/vad_bench/vad_synth.py b/scripts/vad_bench/vad_synth.py new file mode 100755 index 0000000..b27114c --- /dev/null +++ b/scripts/vad_bench/vad_synth.py @@ -0,0 +1,46 @@ +import numpy as np +GR=0.01 # mask grid 10 ms +def mask(spans,n_sec): + m=np.zeros(int(round(n_sec/GR)),bool) + for s,e in spans: + m[int(round(s/GR)):int(round(e/GR))]=True + return m +def trim(u): + fr=len(u)//160; r=np.sqrt((u[:fr*160].reshape(fr,160)**2).mean(1)) + idx=np.where(r>r.max()*0.01)[0] + return u[idx[0]*160:(idx[-1]+1)*160] +def pink(n,rng): + X=np.fft.rfft(rng.standard_normal(n)); f=np.arange(len(X)); f[0]=1 + y=np.fft.irfft(X/np.sqrt(f),n); return y/y.std() +def build(utts,rng): + g=lambda a,b: rng.uniform(a,b) + parts=[];spans=[];t=g(0.5,1.5) + parts.append(rng.standard_normal(int(t*16000))*1e-3) + for i,u in enumerate(utts): + u=trim(u); spans.append((t,t+len(u)/16000)); parts.append(u); t+=len(u)/16000 + gap=g(0.3,2.0) if i1.0: miss+=1 + else: out.append(d*1000) + return out,miss + diff --git a/scripts/vad_bench/wvad.cpp b/scripts/vad_bench/wvad.cpp new file mode 100644 index 0000000..b4c5370 --- /dev/null +++ b/scripts/vad_bench/wvad.cpp @@ -0,0 +1,65 @@ +// Harness over whisper.cpp's public whisper_vad_* API. Not part of whisper.cpp. +// wvad probs ... -> one JSON line per wav (probs + segments) +// wvad time -> detect_speech seconds per rep +#include "whisper.h" +#include +#include +#include +#include +#include +#include +static bool read_wav16(const char* p, std::vector& out) { + FILE* f = fopen(p, "rb"); if (!f) return false; + char id[4]; uint32_t sz; fread(id, 1, 4, f); fread(&sz, 4, 1, f); fread(id, 1, 4, f); + while (fread(id, 1, 4, f) == 4 && fread(&sz, 4, 1, f) == 1) { + if (!memcmp(id, "data", 4)) { + std::vector d(sz / 2); fread(d.data(), 2, d.size(), f); fclose(f); + out.resize(d.size()); for (size_t i = 0; i < d.size(); i++) out[i] = d[i] / 32768.0f; return true; + } + fseek(f, sz, SEEK_CUR); + } + fclose(f); return false; +} +static void segs(whisper_vad_context* v, whisper_vad_params p, const char* key) { + whisper_vad_segments* s = whisper_vad_segments_from_probs(v, p); + printf("\"%s\":[", key); + int n = s ? whisper_vad_segments_n_segments(s) : 0; + for (int i = 0; i < n; i++) + printf("%s[%.2f,%.2f]", i ? "," : "", whisper_vad_segments_get_segment_t0(s, i) / 100.0, whisper_vad_segments_get_segment_t1(s, i) / 100.0); + printf("]"); if (s) whisper_vad_free_segments(s); +} +static void quiet(enum ggml_log_level, const char*, void*) {} +int main(int argc, char** argv) { + whisper_log_set(quiet, nullptr); + if (argc < 5) return 1; + std::string mode = argv[1]; + whisper_vad_context_params cp = whisper_vad_default_context_params(); + cp.n_threads = atoi(argv[3]); cp.use_gpu = false; + whisper_vad_context* v = whisper_vad_init_from_file_with_params(argv[2], cp); + if (!v) { fprintf(stderr, "load failed\n"); return 2; } + std::vector y; + if (mode == "time") { + int reps = atoi(argv[4]); + if (!read_wav16(argv[5], y)) return 3; + for (int r = 0; r < reps; r++) { + auto t0 = std::chrono::steady_clock::now(); + whisper_vad_detect_speech(v, y.data(), (int)y.size()); + double dt = std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); + printf("%.4f\n", dt); + } + return 0; + } + for (int a = 4; a < argc; a++) { + if (!read_wav16(argv[a], y)) return 3; + if (!whisper_vad_detect_speech(v, y.data(), (int)y.size())) return 4; + int n = whisper_vad_n_probs(v); float* pr = whisper_vad_probs(v); + printf("{\"name\":\"%s\",\"n\":%d,\"probs\":[", argv[a], n); + for (int i = 0; i < n; i++) printf("%s%.6f", i ? "," : "", pr[i]); + printf("],"); + segs(v, whisper_vad_default_params(), "default"); printf(","); + whisper_vad_params m = whisper_vad_default_params(); + m.threshold = 0.5f; m.min_speech_duration_ms = 100; m.min_silence_duration_ms = 200; m.speech_pad_ms = 0; + segs(v, m, "matched"); printf("}\n"); + } + whisper_vad_free(v); +} From bc541ef21b835646fc2bc7892d11c4df2ca0154a Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 4 Oct 2026 01:44:26 +0000 Subject: [PATCH 2/4] docs: record that the VAD timing re-run found no quiet machine The load average was checked 19 times over 3 hours and never went below 5, so no timing was run. Keep the earlier speed numbers, each with the load it was measured under, and add the poll log. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] --- docs/vad-benchmarks.md | 15 ++++++++++++- .../results/timing_rerun_load_log.txt | 21 +++++++++++++++++++ 2 files changed, 35 insertions(+), 1 deletion(-) create mode 100644 scripts/vad_bench/results/timing_rerun_load_log.txt diff --git a/docs/vad-benchmarks.md b/docs/vad-benchmarks.md index 0ac1acb..33cd2e7 100644 --- a/docs/vad-benchmarks.md +++ b/docs/vad-benchmarks.md @@ -451,7 +451,20 @@ No number is given here until that measurement is published with its method. ## Timing re-run -Not run yet when this text was written. +A re-run of the timing parts on a quiet machine was tried and did not happen. The +rule was: run only when the 1-minute load average is below 5, check before each run, +poll every 10 minutes for up to 3 hours, serialize with a lock and pin cores. +The machine was shared with other jobs. The load average was checked 19 times, from +22:43 to 01:43, and was never below 5. The lowest value was 6.67 and the highest was +82.37. The poll log is in +[`timing_rerun_load_log.txt`](../scripts/vad_bench/results/timing_rerun_load_log.txt). +No timing was run, and no timing number on this page is new. The speed numbers on this +page are the old ones, taken under the loads that each section states, and they stay +marked as such. The setup for the re-run (a build of `6165e3d`, the 300 s clip, the +models and the driver) was ready, and `speed_silero.py` and `speed_heads.py` run it in +one command each when the machine is quiet. + +The accuracy numbers do not depend on load and were not re-run. ## When to use which diff --git a/scripts/vad_bench/results/timing_rerun_load_log.txt b/scripts/vad_bench/results/timing_rerun_load_log.txt new file mode 100644 index 0000000..504a9a7 --- /dev/null +++ b/scripts/vad_bench/results/timing_rerun_load_log.txt @@ -0,0 +1,21 @@ +22:43:52 start; 22:43:52 up 12 days, 14:48, 1 user, load average: 14.09, 12.73, 10.36 +22:43:52 not quiet: 22:43:52 up 12 days, 14:48, 1 user, load average: 14.09, 12.73, 10.36 +22:53:52 not quiet: 22:53:52 up 12 days, 14:58, 1 user, load average: 38.40, 20.11, 13.44 +23:03:52 not quiet: 23:03:52 up 12 days, 15:08, 1 user, load average: 35.54, 39.78, 28.58 +23:13:52 not quiet: 23:13:52 up 12 days, 15:18, 1 user, load average: 14.53, 19.83, 24.28 +23:23:52 not quiet: 23:23:52 up 12 days, 15:28, 1 user, load average: 34.33, 20.83, 20.40 +23:33:52 not quiet: 23:33:52 up 12 days, 15:38, 1 user, load average: 28.05, 17.41, 17.12 +23:43:52 not quiet: 23:43:52 up 12 days, 15:48, 1 user, load average: 13.25, 12.11, 14.17 +23:53:52 not quiet: 23:53:52 up 12 days, 15:58, 1 user, load average: 82.37, 71.82, 42.46 +00:03:52 not quiet: 00:03:52 up 12 days, 16:08, 1 user, load average: 63.40, 50.75, 45.47 +00:13:52 not quiet: 00:13:52 up 12 days, 16:18, 1 user, load average: 38.12, 53.97, 51.57 +00:23:52 not quiet: 00:23:52 up 12 days, 16:28, 1 user, load average: 32.08, 33.62, 40.68 +00:33:52 not quiet: 00:33:52 up 12 days, 16:38, 1 user, load average: 28.06, 40.43, 41.16 +00:43:52 not quiet: 00:43:52 up 12 days, 16:48, 1 user, load average: 52.52, 43.53, 40.77 +00:53:52 not quiet: 00:53:52 up 12 days, 16:58, 1 user, load average: 7.19, 28.83, 36.45 +01:03:52 not quiet: 01:03:52 up 12 days, 17:08, 1 user, load average: 9.54, 18.41, 27.76 +01:13:52 not quiet: 01:13:52 up 12 days, 17:18, 1 user, load average: 8.63, 17.05, 24.09 +01:23:52 not quiet: 01:23:52 up 12 days, 17:28, 1 user, load average: 6.67, 15.13, 21.99 +01:33:52 not quiet: 01:33:52 up 12 days, 17:38, 1 user, load average: 9.21, 10.77, 16.01 +01:43:52 not quiet: 01:43:52 up 12 days, 17:48, 1 user, load average: 60.44, 35.70, 25.02 +GAVE UP: machine never quiet in 3 h From 9bf6527a8b450d9741295bb510628d6320539035 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 4 Oct 2026 01:46:20 +0000 Subject: [PATCH 3/4] docs: add the slice-only head measurement to the VAD benchmarks Fill the pending section with the load time, peak memory and speed of the stand-alone head slice files (PR 87) against the full ASR files and Silero F16, with the method, the result files and the caveats. Say plainly which numbers were taken on a busy machine. Update the guidance on when to use which detector. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] --- docs/vad-benchmarks.md | 154 +++++-- scripts/vad_bench/README.md | 6 + scripts/vad_bench/results/slice_gate.log | 496 +++++++++++++++++++++ scripts/vad_bench/results/slice_load.json | 218 +++++++++ scripts/vad_bench/results/slice_rss.txt | 18 + scripts/vad_bench/results/slice_sha256.txt | 4 + scripts/vad_bench/results/slice_speed.json | 128 ++++++ scripts/vad_bench/slice_bench.py | 75 ++++ scripts/vad_bench/vad_slice_bench.cpp | 44 ++ 9 files changed, 1109 insertions(+), 34 deletions(-) create mode 100644 scripts/vad_bench/results/slice_gate.log create mode 100644 scripts/vad_bench/results/slice_load.json create mode 100644 scripts/vad_bench/results/slice_rss.txt create mode 100644 scripts/vad_bench/results/slice_sha256.txt create mode 100644 scripts/vad_bench/results/slice_speed.json create mode 100644 scripts/vad_bench/slice_bench.py create mode 100644 scripts/vad_bench/vad_slice_bench.cpp diff --git a/docs/vad-benchmarks.md b/docs/vad-benchmarks.md index 33cd2e7..693c6f6 100644 --- a/docs/vad-benchmarks.md +++ b/docs/vad-benchmarks.md @@ -28,8 +28,11 @@ Accuracy is the frame level F1 (percent) against a coarse reference, see | --- | ---: | ---: | ---: | ---: | ---: | ---: | | Silero (matched settings) | 94.1 | 91.7 | 91.8 | about 660x on 1 thread, `parakeet-cli vad`, includes process start and model load | 60 MB (`parakeet-cli vad`, F32 and F16) | 2.2 MB (F32), 1.3 MB (F16) | | Silero (its own defaults) | 94.4 | 91.9 | 92.1 | not timed separately | same as above | same as above | -| Ultra Q8_0 head | 95.0 | 93.4 | 93.8 | 216x, 8 threads | pending, see below | 941.5 MB (the whole ASR model) | -| Redux packed head | 95.4 | 94.3 | 94.2 | 221x, 8 threads | pending, see below | 213.3 MB (the whole ASR model) | +| Ultra Q8_0 head, full ASR GGUF | 95.0 | 93.4 | 93.8 | 216x, 8 threads (older run); 91.6x on 1 thread and 266.6x on 8 threads in the slice run, busy machine | 905 MiB loaded, 1134 MiB after a 33 s clip | 941.5 MB | +| Ultra Q8_0 head, slice file (PR 87, not released) | same as the full file (see below) | same | same | 91.8x on 1 thread, 257.8x on 8 threads, busy machine | 12 MiB loaded, 241 MiB after a 33 s clip | 6.0 MB | +| Redux packed head, full ASR GGUF | 95.4 | 94.3 | 94.2 | 221x, 8 threads (older run); 94.9x on 1 thread and 267.1x on 8 threads in the slice run, busy machine | 372 MiB loaded, 603 MiB after a 33 s clip | 213.3 MB | +| Redux packed head, slice file (PR 87, not released) | same as the full file | same | same | 95.1x on 1 thread, 266.7x on 8 threads, busy machine | 16 MiB loaded, 246 MiB after a 33 s clip | 9.9 MB | +| Silero F16, same build as the slice run | see Silero rows | | | 736x on 1 thread, 485x on 8 threads, busy machine | 9 MiB loaded, 13 MiB after a 33 s clip | 1.3 MB | | whisper.cpp Silero | 94.1 | not run | not run | in process: 686x on 1 thread | 36 MB (test harness) | 0.9 MB | | ONNX Silero (onnxruntime) | 94.0 | not run | not run | in process: 449x on 1 thread, Python loop | 104 MB | 2.3 MB | @@ -44,17 +47,20 @@ How to read it: - "Matched settings" means Silero ran with the same segmentation settings as the heads (threshold 0.5, minimum speech 100 ms, minimum silence 200 ms, no padding). See [Metrics](#metrics). -- The head speed is the speed of a whole encoder pass on 8 threads, because the head - reads the encoder output. The four head builds that were timed (Ultra Q8_0 216x, - Ultra F16 218x, Redux packed 221x, Redux dequantized F16 222x) are within 3% of - each other. This is a different job from the one Silero does, so the speed columns - of Silero and the heads are not a like for like comparison of the models. They show - the cost of using each one as a stand alone gate. -- Memory of the head as a stand alone VAD is **pending**: it needs a measurement of - the head without the rest of the ASR model, see - [Slice only head](#slice-only-head-pending). -- File size of the heads is the size of the whole ASR GGUF the head lives in. A - stand alone head file is part of the pending measurement. +- Head speeds on 8 threads come from two separate runs on a busy machine (see the + caveats of each section): the older run (216x to 222x for four head builds, a Python + wrapper, load average about 6) and the slice run (255x to 268x, one build of the + C++ harness, load average up to 17 at the start of some rounds). They differ by about + 20%, which is the size of the noise on this machine for 8 threads, not a + difference between methods. The 1 thread rounds of the slice run were within about 5% of each other, + except for two outlier rounds (69.8x and 84.7x). +- The speed columns of Silero and the heads are not a like for like comparison of the + models. They show the cost of using each one as a stand alone gate. +- The heads need the ASR GGUF unless you use a slice file. A slice is a GGUF with only + the head path, made by the code of PR 87 (open, not in a release). Its VAD output is + byte for byte the same as the parent file on three clips, so the accuracy columns + apply to the slice unchanged. See [Slice only head](#slice-only-head). +- All memory and load numbers for the heads are in the slice section. ### Machine and build @@ -433,21 +439,90 @@ One run of `tests/silero_vad_probe` on one CPU thread measured about 40 us per c for F32 and F16 at both rates. That is one run on one machine. It is not a benchmark and is not used above. -## Slice only head (pending) +## Slice only head -Running the head needs the whole ASR GGUF today. A measurement of a head file that -holds only the head and the encoder part it needs ("slice") is being made by another -piece of work. It will give the load time, the peak memory and the speed of the head -as a stand alone detector. +A "slice" is a GGUF that holds only the tensors the VAD path of an Ultra or Redux +file reads. The code that makes it and loads it is PR 87 +(`feat/vad-only-gguf`, open, not in a release). This section compares each full ASR +GGUF with its slice, and with Silero F16, built from the same source tree. -| Quantity | Value | -| --- | --- | -| Peak memory, head alone | pending | -| Load time, head alone | pending | -| Speed, head alone | pending | -| File size, head alone | pending | +### Setup + +- One build of the PR 87 code, one machine (see the table above), the first 300 s of + the talk used elsewhere on this page for speed (`ted300`), and a 33 s clean synthetic + clip for memory. The harness is `scripts/vad_bench/vad_slice_bench.cpp` (a small + program built inside the PR 87 tree, since it uses internal headers) and the driver is + `scripts/vad_bench/slice_bench.py`. +- **Speed:** one timed VAD run after a warm-up run, 5 rounds with the 9 configurations + interleaved, the best of the 5 reported (the median is in the result file), on 1 + thread and on 8 threads. The 8 thread runs were pinned to the 8 idlest cores at the time. + Each round waited for the gate: mean number of other runnable tasks under 4 or the + 1-minute load average under 5. +- **Load time:** the model load alone, median of 10. Warm means the file is in the + page cache (the first load, which warms it, is dropped). Cold means the page cache + for the file was evicted before each load. +- **Peak memory:** peak resident set size from `/usr/bin/time -v`, once per + configuration: loading only, and `parakeet-cli vad` on the 33 s clip with 8 threads. +- **Same output:** `parakeet-cli vad --probabilities` was run with each slice and + with its parent file on three clips (the 33 s clip, a noisy clip, the 600 s talk). + All four slices gave byte-identical JSON on all three clips (12 of 12 pairs). + +### Results + +| Model | Full file | Slice file | Load, warm / cold, full (ms) | Load, warm / cold, slice (ms) | Peak RSS loaded, full / slice (MiB) | Peak RSS after the 33 s clip, full / slice (MiB) | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| Ultra Q8_0 | 941.5 MB | 6.0 MB | 276 / 424 | 0.2 / 2.1 | 905 / 12 | 1134 / 241 | +| Ultra F16 | 1441.9 MB | 9.9 MB | 429 / 666 | 0.4 / 3.5 | 1382 / 15 | 1613 / 246 | +| Redux packed | 213.3 MB | 9.9 MB | 92 / 128 | 0.5 / 3.7 | 372 / 16 | 603 / 246 | +| Redux dequantized F16 | 1441.9 MB | 9.9 MB | 434 / 669 | 0.2 / 3.8 | 1382 / 16 | 1613 / 246 | +| Silero F16 | 1.3 MB | not applicable | 0.4 / 1.5 | not applicable | 9 / not applicable | 13 / not applicable | + +Speed on the 300 s clip, x real time, best of 5 (1 thread / 8 threads): + +| Model | Full file | Slice file | +| --- | ---: | ---: | +| Ultra Q8_0 | 91.6 / 266.6 | 91.8 / 257.8 | +| Ultra F16 | 94.5 / 255.5 | 94.6 / 265.3 | +| Redux packed | 94.9 / 267.1 | 95.1 / 266.7 | +| Redux dequantized F16 | 95.1 / 265.3 | 94.8 / 268.1 | +| Silero F16 | 736.3 / 485.4 | not applicable | + +Medians of the 5 rounds (1 thread / 8 threads) are lower: Ultra Q8_0 full 91.4 / 250.5, +slice 91.5 / 227.8; Redux packed full 93.1 / 252.8, slice 92.4 / 236.3; Silero F16 +702.1 / 473.9. The slice files are 6,007,328 bytes (Ultra Q8_0) and 9,939,488 bytes (the other +three). The SHA-256 of the four slice files is in +[`slice_sha256.txt`](../scripts/vad_bench/results/slice_sha256.txt). + +What this supports: + +- A slice is 6 to 10 MB, loads in under 4 ms, and needs about 12 to 16 MiB loaded and + about 240 to 250 MiB for a 33 s clip. The full file needs 372 MiB to 1.4 GiB just to + load. +- Slice and full file run at the same speed. The differences between them on 1 thread (best of 5) are + 0.1% to 0.3%, far inside the spread of the 5 rounds. +- Silero F16 is smaller (1.3 MB, 13 MiB after the clip), loads in about 0.4 ms, and was + about 7.7 to 8 times faster than the heads on 1 thread (736x against 92x to 95x). + +### Caveats -No number is given here until that measurement is published with its method. +- The 8 thread numbers are noisy. The machine was busy: the gate lines of the speed + rounds in `slice_gate.log` show a 1-minute load average from 4.2 to 17.3 at the start + of the rounds, and some rounds ran at 9 to 17 (the gate also accepts a low runnable + count when the load average is high). The best of 5 on 8 threads is within the noise + of the other values (the 8 thread rounds of one configuration ranged by up to 30%, for + example Ultra Q8_0 full from 185.0x to 266.6x). Do not read a ranking from the 8 thread + column. The 1 thread rounds of one configuration are within about 5% of each other, + except for two outlier rounds (Redux dequantized F16 full 69.8x, slice 84.7x). +- Load times are medians of 10 and tiny for slices, so the warm slice values (0.2 to + 0.5 ms) are close to the timer's resolution and the maximum was up to 2.2 ms. +- Peak memory was measured for a 33 s clip only. The peak memory for a 600 s clip was + not measured, and the memory after the clip grows with the clip length. +- The slice code is in PR 87, which is open and not in a release. GPU backends and the + streaming VAD were not tested with slices. +- These numbers come from one run of the harness. They were not repeated. +- The same output check shows that the probabilities are identical, so the accuracy + tables of this page apply to the slices. It was run on three clips, not on the whole + corpus. ## Timing re-run @@ -458,9 +533,13 @@ The machine was shared with other jobs. The load average was checked 19 times, f 22:43 to 01:43, and was never below 5. The lowest value was 6.67 and the highest was 82.37. The poll log is in [`timing_rerun_load_log.txt`](../scripts/vad_bench/results/timing_rerun_load_log.txt). -No timing was run, and no timing number on this page is new. The speed numbers on this -page are the old ones, taken under the loads that each section states, and they stay -marked as such. The setup for the re-run (a build of `6165e3d`, the 300 s clip, the +No timing was run for this page. The speed numbers of the slice section were taken +separately, on a busy machine, as described there. They are not quiet-machine numbers +either. The speed numbers on this +page are the old ones, taken under the loads that each section states. Numbers taken +on a busy machine: all speeds of the heads against Silero section, all Silero speed +and memory numbers of the parity section, all slice section numbers, and the batched +decode times (load average 2.5 to 18.0 at the start). The setup for the re-run (a build of `6165e3d`, the 300 s clip, the models and the driver) was ready, and `speed_silero.py` and `speed_heads.py` run it in one command each when the machine is quiet. @@ -478,9 +557,15 @@ This is limited to what the numbers above support. model and its scores are close to Silero in the same tests: within about 1.5 F1 points on the synthetic clips at 5 dB and above (Ultra was 0.4 lower in white noise at 5 dB), 2 to 2.5 points higher on the TED talks, and 1.7 to 2.6 points higher at 0 dB pink - noise. It has the higher recall and the lower precision. On these clips its start times are closer to the reference. It - is not a cheap stand alone detector today: it needs the encoder, its speed is about - 220x on 8 threads, and its memory as a stand alone VAD is not yet measured. + noise. It has the higher recall and the lower precision. On these clips its start + times are closer to the reference. +- **The Parakeet head as a stand alone detector.** With a slice file (PR 87, not in a + release) the head is a 6 to 10 MB file that loads in under 4 ms and needs about 240 + to 250 MiB for a 33 s clip, with the same output as the full file. Without it, the head + needs the full ASR GGUF (213 MB to 1.4 GB). Silero is still smaller (1.3 MB), uses less + memory (13 MiB after the clip) and was about 8 times faster per core (736x against + 92x to 95x on 1 thread, on a busy machine). Choose the head for its accuracy on the + tests above, Silero for size and speed. - **Silero from whisper.cpp or from parakeet.cpp.** On the 120 clips and the 600 s talk the two give the same segments (mask F1 99.92) and the same speed within the noise of a shared machine. Choose by the rest of your stack, not by quality or speed. The @@ -497,11 +582,12 @@ lists the inputs, the commands and the Python packages. No audio and no model fi are committed. In short: 1. Fetch the public audio: `fetch_data.py`, `fetch_ted_talks.py`. -2. Heads against Silero: `compare_synthetic.py` then `analyze_synthetic.py`; +2. Slice section: `vad_slice_bench.cpp` (build it inside the PR 87 tree) and `slice_bench.py`. +3. Heads against Silero: `compare_synthetic.py` then `analyze_synthetic.py`; `ted_reference.py` then `compare_ted.py`; `speed_heads.py`. -3. Silero parity: `make_clips.py`, build `wvad.cpp` against whisper.cpp, +4. Silero parity: `make_clips.py`, build `wvad.cpp` against whisper.cpp, `silero_collect.py`, `silero_analyze.py`; speed with `speed_silero.py`. -4. Long talks: `longform_b1.sh`. +5. Long talks: `longform_b1.sh`. Small result files of the runs on this page (tables, per run timings and load logs) are in `scripts/vad_bench/results/`. The raw per clip predictions are not committed; diff --git a/scripts/vad_bench/README.md b/scripts/vad_bench/README.md index dd89a34..7deb9fc 100644 --- a/scripts/vad_bench/README.md +++ b/scripts/vad_bench/README.md @@ -67,3 +67,9 @@ Timing scripts serialize on a lock file (`/tmp/pk-bench.lock`, `flock`), pin cor `taskset`, and record the load average at the start of every run. Run them only on a quiet machine, and keep the load log with the result. Do not quote a timing from a loaded machine as a benchmark. + +## Slice only head (PR 87) + +`vad_slice_bench.cpp` is built inside the PR 87 tree (it uses internal headers) and driven +by `slice_bench.py speed|load|rss`; see its docstring for the environment variables. +Results are in `results/slice_*`. diff --git a/scripts/vad_bench/results/slice_gate.log b/scripts/vad_bench/results/slice_gate.log new file mode 100644 index 0000000..a170abf --- /dev/null +++ b/scripts/vad_bench/results/slice_gate.log @@ -0,0 +1,496 @@ +22:25:57 load ultra-q8_0 full cold=False mean_runnable=22.2 load1=10.02 ok=False +22:26:37 load ultra-q8_0 full cold=False mean_runnable=8.2 load1=9.38 ok=False +22:27:17 load ultra-q8_0 full cold=False mean_runnable=6.0 load1=9.12 ok=False +22:27:57 load ultra-q8_0 full cold=False mean_runnable=13.8 load1=13.77 ok=False +22:28:37 load ultra-q8_0 full cold=False mean_runnable=8.9 load1=11.14 ok=False +22:29:17 load ultra-q8_0 full cold=False mean_runnable=16.6 load1=13.7 ok=False +22:29:57 load ultra-q8_0 full cold=False mean_runnable=5.2 load1=9.73 ok=False +22:30:37 load ultra-q8_0 full cold=False mean_runnable=3.8 load1=6.89 ok=True +22:30:50 load ultra-q8_0 full cold=True mean_runnable=3.3 load1=6.62 ok=True +22:31:05 load ultra-q8_0 slice cold=False mean_runnable=2.6 load1=5.83 ok=True +22:31:15 load ultra-q8_0 slice cold=True mean_runnable=4.2 load1=5.47 ok=False +22:31:55 load ultra-q8_0 slice cold=True mean_runnable=4.5 load1=5.48 ok=False +22:32:35 load ultra-q8_0 slice cold=True mean_runnable=8.7 load1=6.49 ok=False +22:33:15 load ultra-q8_0 slice cold=True mean_runnable=10.5 load1=11.38 ok=False +22:33:55 load ultra-q8_0 slice cold=True mean_runnable=5.6 load1=10.29 ok=False +22:34:35 load ultra-q8_0 slice cold=True mean_runnable=2.1 load1=6.89 ok=True +22:34:45 load ultra-f16 full cold=False mean_runnable=4.1 load1=6.14 ok=False +22:35:25 load ultra-f16 full cold=False mean_runnable=3.2 load1=4.56 ok=True +22:35:40 load ultra-f16 full cold=True mean_runnable=2.5 load1=4.13 ok=True +22:35:57 load ultra-f16 slice cold=False mean_runnable=2.7 load1=3.82 ok=True +22:36:07 load ultra-f16 slice cold=True mean_runnable=2.7 load1=3.46 ok=True +22:36:17 load redux-keep full cold=False mean_runnable=2.3 load1=3.32 ok=True +22:36:28 load redux-keep full cold=True mean_runnable=2.2 load1=3.12 ok=True +22:36:40 load redux-keep slice cold=False mean_runnable=2.5 load1=2.94 ok=True +22:36:50 load redux-keep slice cold=True mean_runnable=2.3 load1=2.72 ok=True +22:37:00 load redux-deq-f16 full cold=False mean_runnable=1.9 load1=2.53 ok=True +22:37:15 load redux-deq-f16 full cold=True mean_runnable=2.1 load1=2.63 ok=True +22:37:32 load redux-deq-f16 slice cold=False mean_runnable=2.8 load1=3.61 ok=True +22:37:42 load redux-deq-f16 slice cold=True mean_runnable=2.6 load1=3.45 ok=True +22:37:52 load silero-vad-f16 cold=False mean_runnable=1.6 load1=3.22 ok=True +22:38:02 load silero-vad-f16 cold=True mean_runnable=2.5 load1=3.04 ok=True +22:38:26 load ultra-q8_0 full cold=False mean_runnable=11.6 load1=5.64 ok=False +22:39:06 load ultra-q8_0 full cold=False mean_runnable=9.6 load1=9.64 ok=False +22:39:46 load ultra-q8_0 full cold=False mean_runnable=3.4 load1=7.5 ok=True +22:39:59 load ultra-q8_0 full cold=True mean_runnable=5.5 load1=9.37 ok=False +22:40:39 load ultra-q8_0 full cold=True mean_runnable=17.2 load1=9.03 ok=False +22:41:19 load ultra-q8_0 full cold=True mean_runnable=27.4 load1=19.5 ok=False +22:41:59 load ultra-q8_0 full cold=True mean_runnable=8.2 load1=17.44 ok=False +22:42:39 load ultra-q8_0 full cold=True mean_runnable=22.4 load1=20.98 ok=False +22:43:19 load ultra-q8_0 full cold=True mean_runnable=14.9 load1=18.05 ok=False +22:43:59 load ultra-q8_0 full cold=True mean_runnable=4.2 load1=13.28 ok=False +22:44:39 load ultra-q8_0 full cold=True mean_runnable=8.7 load1=9.15 ok=False +22:45:19 load ultra-q8_0 full cold=True mean_runnable=3.4 load1=7.49 ok=True +22:45:34 load ultra-q8_0 slice cold=False mean_runnable=4.7 load1=6.71 ok=False +22:46:14 load ultra-q8_0 slice cold=False mean_runnable=10.6 load1=8.53 ok=False +22:46:54 load ultra-q8_0 slice cold=False mean_runnable=9.9 load1=10.03 ok=False +22:47:34 load ultra-q8_0 slice cold=False mean_runnable=4.1 load1=8.36 ok=False +22:48:14 load ultra-q8_0 slice cold=False mean_runnable=17.4 load1=10.4 ok=False +22:48:54 load ultra-q8_0 slice cold=False mean_runnable=17.1 load1=11.38 ok=False +22:49:34 load ultra-q8_0 slice cold=False mean_runnable=4.3 load1=10.02 ok=False +22:50:14 load ultra-q8_0 slice cold=False mean_runnable=3.6 load1=7.13 ok=True +22:50:24 load ultra-q8_0 slice cold=True mean_runnable=6.3 load1=6.65 ok=False +22:51:04 load ultra-q8_0 slice cold=True mean_runnable=6.8 load1=5.5 ok=False +22:51:44 load ultra-q8_0 slice cold=True mean_runnable=35.9 load1=9.91 ok=False +22:52:24 load ultra-q8_0 slice cold=True mean_runnable=50.3 load1=21.91 ok=False +22:53:04 load ultra-q8_0 slice cold=True mean_runnable=29.2 load1=30.68 ok=False +22:53:44 load ultra-q8_0 slice cold=True mean_runnable=50.9 load1=35.89 ok=False +22:54:24 load ultra-q8_0 slice cold=True mean_runnable=54.1 load1=45.19 ok=False +22:55:04 load ultra-q8_0 slice cold=True mean_runnable=15.3 load1=40.41 ok=False +22:55:44 load ultra-q8_0 slice cold=True mean_runnable=57.2 load1=44.67 ok=False +22:56:24 load ultra-q8_0 slice cold=True mean_runnable=56.6 load1=54.46 ok=False +22:57:05 load ultra-q8_0 slice cold=True mean_runnable=62.5 load1=60.04 ok=False +22:57:45 load ultra-q8_0 slice cold=True mean_runnable=52.5 load1=61.9 ok=False +22:58:39 speed round0 t1 mean_runnable=44.2 load1=60.45 ok=False +not quiet, retry in 30s +22:59:19 speed round0 t1 mean_runnable=33.6 load1=52.13 ok=False +not quiet, retry in 30s +22:59:59 speed round0 t1 mean_runnable=14.6 load1=38.61 ok=False +not quiet, retry in 30s +23:00:39 speed round0 t1 mean_runnable=21.9 load1=32.01 ok=False +not quiet, retry in 30s +23:01:19 speed round0 t1 mean_runnable=22.2 load1=27.61 ok=False +not quiet, retry in 30s +23:01:59 speed round0 t1 mean_runnable=52.0 load1=46.33 ok=False +not quiet, retry in 30s +23:02:39 speed round0 t1 mean_runnable=23.1 load1=46.53 ok=False +not quiet, retry in 30s +23:03:19 speed round0 t1 mean_runnable=24.4 load1=41.51 ok=False +not quiet, retry in 30s +23:03:59 speed round0 t1 mean_runnable=18.2 load1=33.9 ok=False +not quiet, retry in 30s +23:04:39 speed round0 t1 mean_runnable=16.6 load1=29.14 ok=False +not quiet, retry in 30s +23:05:20 speed round0 t1 mean_runnable=65.3 load1=32.34 ok=False +not quiet, retry in 30s +23:06:00 speed round0 t1 mean_runnable=50.5 load1=44.47 ok=False +not quiet, retry in 30s +23:06:40 speed round0 t1 mean_runnable=31.5 load1=48.52 ok=False +not quiet, retry in 30s +23:07:20 speed round0 t1 mean_runnable=24.8 load1=36.49 ok=False +not quiet, retry in 30s +23:08:00 speed round0 t1 mean_runnable=20.0 load1=32.66 ok=False +not quiet, retry in 30s +23:08:40 speed round0 t1 mean_runnable=6.3 load1=19.66 ok=False +not quiet, retry in 30s +23:09:20 speed round0 t1 mean_runnable=9.8 load1=13.76 ok=False +not quiet, retry in 30s +23:10:00 speed round0 t1 mean_runnable=11.1 load1=13.02 ok=False +not quiet, retry in 30s +23:10:40 speed round0 t1 mean_runnable=4.7 load1=9.44 ok=False +not quiet, retry in 30s +23:11:20 speed round0 t1 mean_runnable=8.3 load1=8.63 ok=False +not quiet, retry in 30s +23:12:00 speed round0 t1 mean_runnable=8.2 load1=13.15 ok=False +not quiet, retry in 30s +23:12:40 speed round0 t1 mean_runnable=11.8 load1=12.36 ok=False +not quiet, retry in 30s +23:13:20 speed round0 t1 mean_runnable=22.9 load1=14.04 ok=False +not quiet, retry in 30s +23:14:00 speed round0 t1 mean_runnable=6.2 load1=13.68 ok=False +not quiet, retry in 30s +23:14:40 speed round0 t1 mean_runnable=3.9 load1=9.33 ok=True +23:14:51 speed round0 t8 mean_runnable=4.2 load1=7.71 ok=False +not quiet, retry in 30s +23:15:31 speed round0 t8 mean_runnable=4.9 load1=5.74 ok=False +not quiet, retry in 30s +23:16:11 speed round0 t8 mean_runnable=4.0 load1=5.63 ok=False +not quiet, retry in 30s +23:16:51 speed round0 t8 mean_runnable=7.0 load1=6.84 ok=False +not quiet, retry in 30s +23:17:31 speed round0 t8 mean_runnable=4.7 load1=5.52 ok=False +not quiet, retry in 30s +23:18:11 speed round0 t8 mean_runnable=5.7 load1=5.22 ok=False +not quiet, retry in 30s +23:18:51 speed round0 t8 mean_runnable=4.8 load1=5.23 ok=False +not quiet, retry in 30s +23:19:31 speed round0 t8 mean_runnable=8.6 load1=6.58 ok=False +not quiet, retry in 30s +23:20:11 speed round0 t8 mean_runnable=4.0 load1=5.76 ok=True +23:20:23 speed round1 t1 mean_runnable=3.5 load1=5.41 ok=True +23:20:34 speed round1 t8 mean_runnable=4.8 load1=5.66 ok=False +not quiet, retry in 30s +23:21:14 speed round1 t8 mean_runnable=15.0 load1=9.33 ok=False +not quiet, retry in 30s +23:21:54 speed round1 t8 mean_runnable=4.0 load1=15.65 ok=True +23:22:05 speed round2 t1 mean_runnable=14.3 load1=17.87 ok=False +not quiet, retry in 30s +23:22:45 speed round2 t1 mean_runnable=62.0 load1=26.47 ok=False +not quiet, retry in 30s +23:23:26 speed round2 t1 mean_runnable=71.2 load1=45.48 ok=False +not quiet, retry in 30s +23:24:06 speed round2 t1 mean_runnable=11.1 load1=29.91 ok=False +not quiet, retry in 30s +23:24:46 speed round2 t1 mean_runnable=2.4 load1=17.33 ok=True +23:24:57 speed round2 t8 mean_runnable=2.8 load1=14.43 ok=True +23:25:08 speed round3 t1 mean_runnable=3.6 load1=12.66 ok=True +23:25:20 speed round3 t8 mean_runnable=2.4 load1=11.1 ok=True +23:25:31 speed round4 t1 mean_runnable=3.1 load1=9.3 ok=True +23:25:42 speed round4 t8 mean_runnable=2.4 load1=8.26 ok=True +--- rerun speed with local data copy +23:26:32 speed round0 t1 mean_runnable=3.0 load1=5.14 ok=True +23:27:37 speed round0 t8 mean_runnable=2.2 load1=5.01 ok=True +23:28:09 speed round1 t1 mean_runnable=3.2 load1=5.26 ok=True +23:29:16 speed round1 t8 mean_runnable=11.8 load1=7.37 ok=False +not quiet, retry in 30s +23:29:56 speed round1 t8 mean_runnable=19.1 load1=12.11 ok=False +not quiet, retry in 30s +23:30:36 speed round1 t8 mean_runnable=3.4 load1=13.89 ok=True +23:31:09 speed round2 t1 mean_runnable=3.5 load1=12.67 ok=True +23:32:17 speed round2 t8 mean_runnable=37.2 load1=22.05 ok=False +not quiet, retry in 30s +23:32:57 speed round2 t8 mean_runnable=7.2 load1=19.81 ok=False +not quiet, retry in 30s +23:33:37 speed round2 t8 mean_runnable=39.3 load1=23.89 ok=False +not quiet, retry in 30s +23:34:17 speed round2 t8 mean_runnable=5.8 load1=24.75 ok=False +not quiet, retry in 30s +23:34:57 speed round2 t8 mean_runnable=10.6 load1=16.44 ok=False +not quiet, retry in 30s +23:35:37 speed round2 t8 mean_runnable=17.6 load1=19.24 ok=False +not quiet, retry in 30s +23:36:17 speed round2 t8 mean_runnable=3.8 load1=12.7 ok=True +23:36:51 speed round3 t1 mean_runnable=3.7 load1=11.9 ok=True +23:37:58 speed round3 t8 mean_runnable=8.5 load1=9.58 ok=False +not quiet, retry in 30s +23:38:38 speed round3 t8 mean_runnable=9.4 load1=8.78 ok=False +not quiet, retry in 30s +23:39:18 speed round3 t8 mean_runnable=4.2 load1=8.13 ok=False +not quiet, retry in 30s +23:39:58 speed round3 t8 mean_runnable=7.7 load1=7.36 ok=False +not quiet, retry in 30s +23:40:38 speed round3 t8 mean_runnable=11.0 load1=6.69 ok=False +not quiet, retry in 30s +23:41:18 speed round3 t8 mean_runnable=5.0 load1=7.02 ok=False +not quiet, retry in 30s +23:41:58 speed round3 t8 mean_runnable=22.2 load1=13.86 ok=False +not quiet, retry in 30s +23:42:38 speed round3 t8 mean_runnable=9.4 load1=13.08 ok=False +not quiet, retry in 30s +23:43:18 speed round3 t8 mean_runnable=12.7 load1=13.06 ok=False +not quiet, retry in 30s +23:43:58 speed round3 t8 mean_runnable=7.4 load1=12.51 ok=False +not quiet, retry in 30s +23:44:38 speed round3 t8 mean_runnable=12.9 load1=12.55 ok=False +not quiet, retry in 30s +23:45:18 speed round3 t8 mean_runnable=10.8 load1=11.65 ok=False +not quiet, retry in 30s +23:45:58 speed round3 t8 mean_runnable=17.6 load1=14.55 ok=False +not quiet, retry in 30s +23:46:38 speed round3 t8 mean_runnable=30.9 load1=22.36 ok=False +not quiet, retry in 30s +23:47:18 speed round3 t8 mean_runnable=23.1 load1=25.02 ok=False +not quiet, retry in 30s +23:47:58 speed round3 t8 mean_runnable=83.7 load1=43.53 ok=False +not quiet, retry in 30s +23:48:38 speed round3 t8 mean_runnable=119.0 load1=67.52 ok=False +not quiet, retry in 30s +23:49:18 speed round3 t8 mean_runnable=90.1 load1=103.67 ok=False +not quiet, retry in 30s +23:49:58 speed round3 t8 mean_runnable=113.4 load1=96.99 ok=False +not quiet, retry in 30s +23:50:38 speed round3 t8 mean_runnable=119.6 load1=119.77 ok=False +not quiet, retry in 30s +23:51:18 speed round3 t8 mean_runnable=53.0 load1=102.39 ok=False +not quiet, retry in 30s +23:51:58 speed round3 t8 mean_runnable=61.3 load1=87.49 ok=False +not quiet, retry in 30s +23:52:38 speed round3 t8 mean_runnable=51.3 load1=88.8 ok=False +not quiet, retry in 30s +23:53:18 speed round3 t8 mean_runnable=72.8 load1=88.13 ok=False +not quiet, retry in 30s +23:53:58 speed round3 t8 mean_runnable=65.3 load1=83.06 ok=False +not quiet, retry in 30s +23:54:38 speed round3 t8 mean_runnable=80.5 load1=82.42 ok=False +not quiet, retry in 30s +23:55:18 speed round3 t8 mean_runnable=76.4 load1=89.03 ok=False +not quiet, retry in 30s +23:55:58 speed round3 t8 mean_runnable=46.8 load1=80.18 ok=False +not quiet, retry in 30s +23:56:38 speed round3 t8 mean_runnable=14.1 load1=61.31 ok=False +not quiet, retry in 30s +23:57:18 speed round3 t8 mean_runnable=18.6 load1=42.61 ok=False +not quiet, retry in 30s +23:57:58 speed round3 t8 mean_runnable=24.1 load1=31.22 ok=False +not quiet, retry in 30s +23:58:38 speed round3 t8 mean_runnable=30.4 load1=27.78 ok=False +not quiet, retry in 30s +23:59:18 speed round3 t8 mean_runnable=83.2 load1=52.49 ok=False +not quiet, retry in 30s +23:59:58 speed round3 t8 mean_runnable=41.8 load1=48.26 ok=False +not quiet, retry in 30s +00:00:38 speed round3 t8 mean_runnable=72.8 load1=55.97 ok=False +not quiet, retry in 30s +00:01:18 speed round3 t8 mean_runnable=10.9 load1=46.46 ok=False +not quiet, retry in 30s +00:01:58 speed round3 t8 mean_runnable=16.4 load1=31.58 ok=False +not quiet, retry in 30s +00:02:38 speed round3 t8 mean_runnable=18.3 load1=25.8 ok=False +not quiet, retry in 30s +00:03:18 speed round3 t8 mean_runnable=61.1 load1=26.92 ok=False +not quiet, retry in 30s +00:03:58 speed round3 t8 mean_runnable=108.9 load1=65.61 ok=False +not quiet, retry in 30s +00:04:38 speed round3 t8 mean_runnable=78.7 load1=66.31 ok=False +not quiet, retry in 30s +00:05:18 speed round3 t8 mean_runnable=40.3 load1=64.74 ok=False +not quiet, retry in 30s +00:05:58 speed round3 t8 mean_runnable=32.8 load1=52.31 ok=False +not quiet, retry in 30s +00:06:38 speed round3 t8 mean_runnable=56.0 load1=49.43 ok=False +not quiet, retry in 30s +00:07:18 speed round3 t8 mean_runnable=60.2 load1=60.68 ok=False +not quiet, retry in 30s +00:07:58 speed round3 t8 mean_runnable=113.5 load1=81.87 ok=False +not quiet, retry in 30s +00:08:38 speed round3 t8 mean_runnable=64.5 load1=80.12 ok=False +not quiet, retry in 30s +00:09:18 speed round3 t8 mean_runnable=52.6 load1=71.99 ok=False +not quiet, retry in 30s +00:09:58 speed round3 t8 mean_runnable=51.6 load1=65.16 ok=False +not quiet, retry in 30s +00:10:38 speed round3 t8 mean_runnable=79.2 load1=72.1 ok=False +not quiet, retry in 30s +00:11:18 speed round3 t8 mean_runnable=47.8 load1=68.7 ok=False +not quiet, retry in 30s +00:11:58 speed round3 t8 mean_runnable=29.4 load1=54.79 ok=False +not quiet, retry in 30s +00:12:38 speed round3 t8 mean_runnable=36.2 load1=45.03 ok=False +not quiet, retry in 30s +00:13:18 speed round3 t8 mean_runnable=46.0 load1=50.37 ok=False +not quiet, retry in 30s +00:13:58 speed round3 t8 mean_runnable=15.9 load1=37.31 ok=False +not quiet, retry in 30s +00:14:38 speed round3 t8 mean_runnable=12.8 load1=24.53 ok=False +not quiet, retry in 30s +00:15:18 speed round3 t8 mean_runnable=9.1 load1=20.61 ok=False +not quiet, retry in 30s +00:15:58 speed round3 t8 mean_runnable=26.6 load1=23.46 ok=False +not quiet, retry in 30s +00:16:38 speed round3 t8 mean_runnable=44.2 load1=29.85 ok=False +not quiet, retry in 30s +00:17:18 speed round3 t8 mean_runnable=20.3 load1=22.12 ok=False +not quiet, retry in 30s +00:17:58 speed round3 t8 mean_runnable=10.2 load1=23.32 ok=False +not quiet, retry in 30s +00:18:38 speed round3 t8 mean_runnable=20.4 load1=19.81 ok=False +not quiet, retry in 30s +00:19:18 speed round3 t8 mean_runnable=33.9 load1=18.64 ok=False +not quiet, retry in 30s +00:19:58 speed round3 t8 mean_runnable=38.0 load1=33.8 ok=False +not quiet, retry in 30s +00:20:38 speed round3 t8 mean_runnable=36.8 load1=34.77 ok=False +not quiet, retry in 30s +00:21:18 speed round3 t8 mean_runnable=29.9 load1=38.84 ok=False +not quiet, retry in 30s +00:21:58 speed round3 t8 mean_runnable=34.6 load1=41.54 ok=False +not quiet, retry in 30s +00:22:38 speed round3 t8 mean_runnable=27.6 load1=36.55 ok=False +not quiet, retry in 30s +00:23:18 speed round3 t8 mean_runnable=13.7 load1=27.11 ok=False +not quiet, retry in 30s +00:23:58 speed round3 t8 mean_runnable=37.2 load1=35.28 ok=False +not quiet, retry in 30s +00:24:38 speed round3 t8 mean_runnable=29.1 load1=48.33 ok=False +not quiet, retry in 30s +00:25:18 speed round3 t8 mean_runnable=5.2 load1=32.66 ok=False +not quiet, retry in 30s +00:25:58 speed round3 t8 mean_runnable=5.4 load1=21.01 ok=False +not quiet, retry in 30s +00:26:38 speed round3 t8 mean_runnable=63.4 load1=20.82 ok=False +not quiet, retry in 30s +00:27:18 speed round3 t8 mean_runnable=98.2 load1=35.79 ok=False +not quiet, retry in 30s +00:27:58 speed round3 t8 mean_runnable=10.6 load1=28.54 ok=False +not quiet, retry in 30s +00:28:38 speed round3 t8 mean_runnable=26.8 load1=40.58 ok=False +not quiet, retry in 30s +00:29:18 speed round3 t8 mean_runnable=93.5 load1=58.46 ok=False +not quiet, retry in 30s +00:29:58 speed round3 t8 mean_runnable=11.2 load1=53.45 ok=False +not quiet, retry in 30s +00:30:38 speed round3 t8 mean_runnable=35.5 load1=37.22 ok=False +not quiet, retry in 30s +00:31:18 speed round3 t8 mean_runnable=92.5 load1=57.18 ok=False +not quiet, retry in 30s +00:31:58 speed round3 t8 mean_runnable=89.3 load1=72.14 ok=False +not quiet, retry in 30s +00:32:38 speed round3 t8 mean_runnable=10.2 load1=50.99 ok=False +not quiet, retry in 30s +00:33:18 speed round3 t8 mean_runnable=12.2 load1=37.38 ok=False +not quiet, retry in 30s +00:33:58 speed round3 t8 mean_runnable=15.9 load1=26.78 ok=False +not quiet, retry in 30s +00:34:38 speed round3 t8 mean_runnable=21.4 load1=23.94 ok=False +not quiet, retry in 30s +00:35:18 speed round3 t8 mean_runnable=17.2 load1=23.2 ok=False +not quiet, retry in 30s +00:35:58 speed round3 t8 mean_runnable=51.4 load1=32.42 ok=False +not quiet, retry in 30s +00:36:38 speed round3 t8 mean_runnable=8.5 load1=29.3 ok=False +not quiet, retry in 30s +00:37:18 speed round3 t8 mean_runnable=23.8 load1=23.32 ok=False +not quiet, retry in 30s +00:37:58 speed round3 t8 mean_runnable=58.1 load1=31.52 ok=False +not quiet, retry in 30s +00:38:38 speed round3 t8 mean_runnable=67.2 load1=47.1 ok=False +not quiet, retry in 30s +00:39:18 speed round3 t8 mean_runnable=19.9 load1=45.81 ok=False +not quiet, retry in 30s +00:39:58 speed round3 t8 mean_runnable=27.9 load1=39.97 ok=False +not quiet, retry in 30s +00:40:38 speed round3 t8 mean_runnable=23.8 load1=31.28 ok=False +not quiet, retry in 30s +00:41:18 speed round3 t8 mean_runnable=30.2 load1=22.59 ok=False +not quiet, retry in 30s +00:41:58 speed round3 t8 mean_runnable=70.0 load1=40.36 ok=False +not quiet, retry in 30s +00:42:38 speed round3 t8 mean_runnable=61.5 load1=54.21 ok=False +not quiet, retry in 30s +00:43:18 speed round3 t8 mean_runnable=39.9 load1=59.64 ok=False +not quiet, retry in 30s +00:43:58 speed round3 t8 mean_runnable=50.9 load1=52.32 ok=False +not quiet, retry in 30s +00:44:38 speed round3 t8 mean_runnable=35.5 load1=48.34 ok=False +not quiet, retry in 30s +00:45:18 speed round3 t8 mean_runnable=11.1 load1=34.3 ok=False +not quiet, retry in 30s +00:45:58 speed round3 t8 mean_runnable=8.1 load1=22.41 ok=False +not quiet, retry in 30s +00:46:39 speed round3 t8 mean_runnable=89.8 load1=34.07 ok=False +not quiet, retry in 30s +00:47:19 speed round3 t8 mean_runnable=21.6 load1=27.81 ok=False +not quiet, retry in 30s +00:47:59 speed round3 t8 mean_runnable=55.2 load1=45.27 ok=False +not quiet, retry in 30s +00:48:39 speed round3 t8 mean_runnable=53.6 load1=62.55 ok=False +not quiet, retry in 30s +00:49:19 speed round3 t8 mean_runnable=90.2 load1=67.05 ok=False +not quiet, retry in 30s +00:49:59 speed round3 t8 mean_runnable=42.9 load1=56.68 ok=False +not quiet, retry in 30s +00:50:39 speed round3 t8 mean_runnable=69.0 load1=55.88 ok=False +not quiet, retry in 30s +00:51:19 speed round3 t8 mean_runnable=24.2 load1=39.1 ok=False +not quiet, retry in 30s +00:51:59 speed round3 t8 mean_runnable=4.6 load1=22.81 ok=False +not quiet, retry in 30s +00:52:39 speed round3 t8 mean_runnable=10.3 load1=13.88 ok=False +not quiet, retry in 30s +00:53:19 speed round3 t8 mean_runnable=5.5 load1=9.36 ok=False +not quiet, retry in 30s +00:53:59 speed round3 t8 mean_runnable=5.2 load1=6.94 ok=False +not quiet, retry in 30s +00:54:39 speed round3 t8 mean_runnable=11.3 load1=6.98 ok=False +not quiet, retry in 30s +00:55:19 speed round3 t8 mean_runnable=13.4 load1=9.84 ok=False +not quiet, retry in 30s +00:55:59 speed round3 t8 mean_runnable=3.7 load1=9.3 ok=True +00:56:34 speed round4 t1 mean_runnable=32.6 load1=18.42 ok=False +not quiet, retry in 30s +00:57:14 speed round4 t1 mean_runnable=30.8 load1=28.52 ok=False +not quiet, retry in 30s +00:57:54 speed round4 t1 mean_runnable=10.0 load1=33.09 ok=False +not quiet, retry in 30s +00:58:34 speed round4 t1 mean_runnable=7.3 load1=20.45 ok=False +not quiet, retry in 30s +00:59:14 speed round4 t1 mean_runnable=36.8 load1=24.39 ok=False +not quiet, retry in 30s +00:59:54 speed round4 t1 mean_runnable=17.9 load1=17.4 ok=False +not quiet, retry in 30s +01:00:34 speed round4 t1 mean_runnable=67.2 load1=21.99 ok=False +not quiet, retry in 30s +01:01:14 speed round4 t1 mean_runnable=8.2 load1=26.74 ok=False +not quiet, retry in 30s +01:01:54 speed round4 t1 mean_runnable=22.9 load1=20.96 ok=False +not quiet, retry in 30s +01:02:34 speed round4 t1 mean_runnable=6.0 load1=13.64 ok=False +not quiet, retry in 30s +01:03:14 speed round4 t1 mean_runnable=4.6 load1=11.96 ok=False +not quiet, retry in 30s +01:03:54 speed round4 t1 mean_runnable=10.8 load1=9.54 ok=False +not quiet, retry in 30s +01:04:34 speed round4 t1 mean_runnable=24.8 load1=22.15 ok=False +not quiet, retry in 30s +01:05:14 speed round4 t1 mean_runnable=105.2 load1=43.74 ok=False +not quiet, retry in 30s +01:05:54 speed round4 t1 mean_runnable=8.0 load1=40.64 ok=False +not quiet, retry in 30s +01:06:34 speed round4 t1 mean_runnable=8.0 load1=26.94 ok=False +not quiet, retry in 30s +01:07:14 speed round4 t1 mean_runnable=8.7 load1=16.64 ok=False +not quiet, retry in 30s +01:07:54 speed round4 t1 mean_runnable=13.6 load1=13.63 ok=False +not quiet, retry in 30s +01:08:34 speed round4 t1 mean_runnable=23.4 load1=14.55 ok=False +not quiet, retry in 30s +01:09:15 speed round4 t1 mean_runnable=23.7 load1=23.17 ok=False +not quiet, retry in 30s +01:09:55 speed round4 t1 mean_runnable=21.6 load1=25.05 ok=False +not quiet, retry in 30s +01:10:35 speed round4 t1 mean_runnable=41.0 load1=28.11 ok=False +not quiet, retry in 30s +01:11:15 speed round4 t1 mean_runnable=10.7 load1=20.96 ok=False +not quiet, retry in 30s +01:11:55 speed round4 t1 mean_runnable=5.5 load1=18.81 ok=False +not quiet, retry in 30s +01:12:35 speed round4 t1 mean_runnable=5.8 load1=13.02 ok=False +not quiet, retry in 30s +01:13:15 speed round4 t1 mean_runnable=7.2 load1=9.47 ok=False +not quiet, retry in 30s +01:13:55 speed round4 t1 mean_runnable=14.8 load1=8.18 ok=False +not quiet, retry in 30s +01:14:35 speed round4 t1 mean_runnable=56.4 load1=18.69 ok=False +not quiet, retry in 30s +01:15:15 speed round4 t1 mean_runnable=122.5 load1=58.63 ok=False +not quiet, retry in 30s +01:15:55 speed round4 t1 mean_runnable=17.6 load1=46.06 ok=False +not quiet, retry in 30s +01:16:35 speed round4 t1 mean_runnable=14.4 load1=46.05 ok=False +not quiet, retry in 30s +01:17:15 speed round4 t1 mean_runnable=26.8 load1=37.02 ok=False +not quiet, retry in 30s +01:17:55 speed round4 t1 mean_runnable=9.6 load1=23.92 ok=False +not quiet, retry in 30s +01:18:35 speed round4 t1 mean_runnable=4.5 load1=16.83 ok=False +not quiet, retry in 30s +01:19:15 speed round4 t1 mean_runnable=6.3 load1=11.49 ok=False +not quiet, retry in 30s +01:19:55 speed round4 t1 mean_runnable=18.8 load1=17.0 ok=False +not quiet, retry in 30s +01:20:35 speed round4 t1 mean_runnable=9.4 load1=20.12 ok=False +not quiet, retry in 30s +01:21:15 speed round4 t1 mean_runnable=9.3 load1=14.33 ok=False +not quiet, retry in 30s +01:21:55 speed round4 t1 mean_runnable=6.3 load1=10.56 ok=False +not quiet, retry in 30s +01:22:35 speed round4 t1 mean_runnable=5.5 load1=7.38 ok=False +not quiet, retry in 30s +01:23:15 speed round4 t1 mean_runnable=5.5 load1=6.77 ok=False +not quiet, retry in 30s +01:23:55 speed round4 t1 mean_runnable=4.2 load1=6.94 ok=False +not quiet, retry in 30s +01:24:35 speed round4 t1 mean_runnable=4.7 load1=5.15 ok=False +not quiet, retry in 30s +01:25:15 speed round4 t1 mean_runnable=3.2 load1=4.54 ok=True +01:26:20 speed round4 t8 mean_runnable=8.2 load1=4.15 ok=True diff --git a/scripts/vad_bench/results/slice_load.json b/scripts/vad_bench/results/slice_load.json new file mode 100644 index 0000000..a157216 --- /dev/null +++ b/scripts/vad_bench/results/slice_load.json @@ -0,0 +1,218 @@ +{ + "ultra-q8_0 full|warm": [ + 271.7, + 269.9, + 280.0, + 269.6, + 270.3, + 277.3, + 274.7, + 277.0, + 300.8, + 279.3 + ], + "ultra-q8_0 full|cold": [ + 451.1, + 440.4, + 460.9, + 427.3, + 423.8, + 419.5, + 420.4, + 416.2, + 419.5, + 424.3 + ], + "ultra-q8_0 slice|warm": [ + 1.4, + 0.5, + 0.4, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2 + ], + "ultra-q8_0 slice|cold": [ + 5.1, + 2.9, + 2.3, + 2.2, + 1.9, + 2.1, + 2.1, + 2.0, + 2.1, + 2.0 + ], + "ultra-f16 full|warm": [ + 430.2, + 428.6, + 425.2, + 425.8, + 426.9, + 430.0, + 435.2, + 435.0, + 439.2, + 426.8 + ], + "ultra-f16 full|cold": [ + 678.2, + 684.3, + 664.1, + 666.3, + 660.9, + 672.9, + 664.9, + 658.9, + 674.0, + 664.7 + ], + "ultra-f16 slice|warm": [ + 2.2, + 0.8, + 0.6, + 0.5, + 0.4, + 0.4, + 0.3, + 0.3, + 0.4, + 0.3 + ], + "ultra-f16 slice|cold": [ + 6.7, + 4.2, + 3.3, + 3.6, + 3.6, + 3.2, + 3.2, + 3.4, + 3.6, + 3.4 + ], + "redux-keep full|warm": [ + 94.2, + 88.1, + 86.7, + 90.4, + 93.3, + 93.3, + 91.8, + 92.1, + 90.3, + 93.0 + ], + "redux-keep full|cold": [ + 145.0, + 130.6, + 129.9, + 135.2, + 122.3, + 126.0, + 128.3, + 128.1, + 122.2, + 125.7 + ], + "redux-keep slice|warm": [ + 2.2, + 0.8, + 0.6, + 0.5, + 0.4, + 0.4, + 0.4, + 0.4, + 0.5, + 0.4 + ], + "redux-keep slice|cold": [ + 6.9, + 4.4, + 3.5, + 3.7, + 3.7, + 3.6, + 3.9, + 4.1, + 3.7, + 3.5 + ], + "redux-deq-f16 full|warm": [ + 439.0, + 442.8, + 443.6, + 432.5, + 429.8, + 434.9, + 432.5, + 436.7, + 433.8, + 425.7 + ], + "redux-deq-f16 full|cold": [ + 659.3, + 682.1, + 665.0, + 677.6, + 667.6, + 666.2, + 659.4, + 686.0, + 706.4, + 670.0 + ], + "redux-deq-f16 slice|warm": [ + 1.7, + 0.5, + 0.3, + 0.3, + 0.2, + 0.2, + 0.2, + 0.3, + 0.2, + 0.2 + ], + "redux-deq-f16 slice|cold": [ + 6.7, + 4.6, + 3.8, + 3.8, + 3.8, + 3.7, + 3.7, + 3.8, + 3.7, + 3.9 + ], + "silero-vad-f16|warm": [ + 0.9, + 0.5, + 0.4, + 0.4, + 0.4, + 0.4, + 0.4, + 0.4, + 0.4, + 0.4 + ], + "silero-vad-f16|cold": [ + 3.8, + 2.6, + 1.5, + 1.5, + 1.5, + 1.5, + 1.5, + 1.6, + 1.6, + 1.5 + ] +} \ No newline at end of file diff --git a/scripts/vad_bench/results/slice_rss.txt b/scripts/vad_bench/results/slice_rss.txt new file mode 100644 index 0000000..6afa114 --- /dev/null +++ b/scripts/vad_bench/results/slice_rss.txt @@ -0,0 +1,18 @@ +ultra-q8_0 full load-only peak RSS 905.1 MiB +ultra-q8_0 full load+vad33s peak RSS 1133.4 MiB +ultra-q8_0 slice load-only peak RSS 11.7 MiB +ultra-q8_0 slice load+vad33s peak RSS 240.7 MiB +ultra-f16 full load-only peak RSS 1382.4 MiB +ultra-f16 full load+vad33s peak RSS 1612.3 MiB +ultra-f16 slice load-only peak RSS 15.5 MiB +ultra-f16 slice load+vad33s peak RSS 245.5 MiB +redux-keep full load-only peak RSS 372.1 MiB +redux-keep full load+vad33s peak RSS 602.7 MiB +redux-keep slice load-only peak RSS 15.5 MiB +redux-keep slice load+vad33s peak RSS 245.9 MiB +redux-deq-f16 full load-only peak RSS 1382.4 MiB +redux-deq-f16 full load+vad33s peak RSS 1612.4 MiB +redux-deq-f16 slice load-only peak RSS 15.5 MiB +redux-deq-f16 slice load+vad33s peak RSS 246.4 MiB +silero-vad-f16 load-only peak RSS 8.9 MiB +silero-vad-f16 load+vad33s peak RSS 13.3 MiB diff --git a/scripts/vad_bench/results/slice_sha256.txt b/scripts/vad_bench/results/slice_sha256.txt new file mode 100644 index 0000000..7f442a9 --- /dev/null +++ b/scripts/vad_bench/results/slice_sha256.txt @@ -0,0 +1,4 @@ +1b8d46253e2d5205e5058abb7ff12769046001067ddc4892707180cb7a7e22fb redux-deq-f16-vad.gguf +588e1d6e2ee5b6cdfd9ec5ea98dc0993d5bea498d9cc4ec8d6077041eef8a34f redux-keep-vad.gguf +f9df4202f38c528819e663435b21654f44d841f88a0f03d8f48abbb3c00e963e ultra-f16-vad.gguf +8b891a4435e97438104ca07c72530d0c5fe62b986baee48b2dd4e1500c1d4758 ultra-q8_0-vad.gguf diff --git a/scripts/vad_bench/results/slice_speed.json b/scripts/vad_bench/results/slice_speed.json new file mode 100644 index 0000000..99ff2c6 --- /dev/null +++ b/scripts/vad_bench/results/slice_speed.json @@ -0,0 +1,128 @@ +{ + "ultra-q8_0 full|1": [ + 91.6, + 91.6, + 91.0, + 89.4, + 91.4 + ], + "ultra-q8_0 full|8": [ + 266.6, + 251.7, + 250.5, + 185.0, + 203.0 + ], + "ultra-q8_0 slice|1": [ + 91.2, + 91.5, + 91.6, + 89.4, + 91.8 + ], + "ultra-q8_0 slice|8": [ + 257.8, + 248.7, + 222.2, + 227.8, + 221.0 + ], + "ultra-f16 full|1": [ + 93.5, + 92.6, + 91.5, + 92.4, + 94.5 + ], + "ultra-f16 full|8": [ + 253.8, + 255.5, + 233.9, + 255.3, + 254.2 + ], + "ultra-f16 slice|1": [ + 94.3, + 92.5, + 91.9, + 88.5, + 94.6 + ], + "ultra-f16 slice|8": [ + 257.7, + 204.4, + 251.1, + 265.3, + 229.9 + ], + "redux-keep full|1": [ + 94.1, + 92.1, + 91.0, + 93.1, + 94.9 + ], + "redux-keep full|8": [ + 252.8, + 264.0, + 251.3, + 226.8, + 267.1 + ], + "redux-keep slice|1": [ + 94.7, + 90.4, + 92.4, + 91.7, + 95.1 + ], + "redux-keep slice|8": [ + 266.7, + 265.6, + 219.3, + 181.2, + 236.3 + ], + "redux-deq-f16 full|1": [ + 95.1, + 89.8, + 69.8, + 88.4, + 94.0 + ], + "redux-deq-f16 full|8": [ + 259.7, + 254.5, + 264.6, + 265.3, + 260.6 + ], + "redux-deq-f16 slice|1": [ + 94.8, + 88.9, + 84.7, + 89.5, + 93.4 + ], + "redux-deq-f16 slice|8": [ + 268.1, + 261.9, + 263.2, + 263.3, + 260.1 + ], + "silero-vad-f16|1": [ + 736.3, + 724.6, + 661.2, + 687.6, + 702.1 + ], + "silero-vad-f16|8": [ + 473.9, + 483.3, + 459.1, + 485.4, + 462.2 + ] +} \ No newline at end of file diff --git a/scripts/vad_bench/slice_bench.py b/scripts/vad_bench/slice_bench.py new file mode 100644 index 0000000..2f86f85 --- /dev/null +++ b/scripts/vad_bench/slice_bench.py @@ -0,0 +1,75 @@ +#!/usr/bin/env python3 +"""Stand-alone head (slice) benchmark: speed, load time and peak memory, full ASR GGUF vs slice GGUF. + +usage: slice_bench.py speed|load|rss +Environment: GGUF_DIR (full GGUFs: ultra-q8_0 ultra-f16 redux-keep redux-deq-f16, plus silero-vad-f16.gguf), +SLICE_DIR (the *-vad.gguf slices of PR 87), WAV300 (300 s clip), CLIP (a short clip for the memory run), +VADBENCH (binary built from vad_slice_bench.cpp inside the PR 87 tree), CLI (parakeet-cli), OUT (output dir). +""" +import subprocess, time, statistics, json, sys, os, re +G=os.environ["GGUF_DIR"]; V=os.environ["SLICE_DIR"]; B=os.environ["VADBENCH"]; WAV=os.environ["WAV300"] +CLIP=os.environ["CLIP"]; CLI=os.environ["CLI"]; OUT=os.environ.get("OUT",".") +S=G +CFG=[("ultra-q8_0 full",G+"/ultra-q8_0.gguf"),("ultra-q8_0 slice",V+"/ultra-q8_0-vad.gguf"), + ("ultra-f16 full",G+"/ultra-f16.gguf"),("ultra-f16 slice",V+"/ultra-f16-vad.gguf"), + ("redux-keep full",G+"/redux-keep.gguf"),("redux-keep slice",V+"/redux-keep-vad.gguf"), + ("redux-deq-f16 full",G+"/redux-deq-f16.gguf"),("redux-deq-f16 slice",V+"/redux-deq-f16-vad.gguf"), + ("silero-vad-f16",G+"/silero-vad-f16.gguf")] +log=open(OUT+"/gate.log","a") +def cpu_idle(): + r={} + for l in open("/proc/stat"): + if l.startswith("cpu") and l[3].isdigit(): + f=l.split(); v=list(map(int,f[1:])); r[int(f[0][3:])]=(v[3]+v[4],sum(v)) + return r +def gate(tag): + a=cpu_idle(); runs=[] + for _ in range(20): + runs.append(int(open("/proc/loadavg").read().split()[3].split("/")[0])-1); time.sleep(0.5) + b=cpu_idle(); load1=float(open("/proc/loadavg").read().split()[0]) + busy={c:1-(b[c][0]-a[c][0])/max(1,b[c][1]-a[c][1]) for c in a} + order=sorted(busy,key=busy.get) + m=statistics.mean(runs); ok=(m<4) or (load1<5) + log.write(f"{time.strftime('%H:%M:%S')} {tag} mean_runnable={m:.1f} load1={load1} ok={ok} idlest={order[:8]}\n"); log.flush() + return ok,order +def runb(args,cpus): + cmd=["flock","/tmp/pk-bench.lock","taskset","-c",",".join(map(str,cpus))]+args + return subprocess.run(cmd,capture_output=True,text=True,check=True).stdout.split() +mode=sys.argv[1] +if mode=="speed": + res={(n,t):[] for n,_ in CFG for t in (1,8)} + for rnd in range(5): + for t in (1,8): + while True: + ok,order=gate(f"speed round{rnd} t{t}") + if ok: break + log.write("not quiet, retry in 30s\n"); log.flush(); time.sleep(30) + cpus=order[:t] + for n,p in CFG: + s,x=runb([B,"speed",p,WAV,str(t)],cpus) + res[(n,t)].append(float(x)) + out={f"{n}|{t}":v for (n,t),v in res.items()} + json.dump(out,open(OUT+"/speed.json","w"),indent=1) + for (n,t),v in res.items(): print(f"{n:22s} {t} thr best {max(v):7.1f}x median {statistics.median(v):7.1f}x all {v}") +elif mode=="load": + out={} + for n,p in CFG: + if len(sys.argv)>2 and not n.startswith(sys.argv[2]): continue + for cold in (False,True): + while True: + ok,order=gate(f"load {n} cold={cold}") + if ok: break + time.sleep(30) + a=[B,"load",p,"10"]+(["cold"] if cold else []) + if not cold: a=[B,"load",p,"11"] # first load warms the page cache and is dropped + v=list(map(float,runb(a,order[:8]))) + if not cold: v=v[1:] + out[f"{n}|{'cold' if cold else 'warm'}"]=v + print(f"{n:22s} {'cold' if cold else 'warm'} median {statistics.median(v):8.1f} ms min {min(v):.1f} max {max(v):.1f}",flush=True) + json.dump(out,open(OUT+"/load%s.json"%("-"+sys.argv[2] if len(sys.argv)>2 else ""),"w"),indent=1) +elif mode=="rss": + for n,p in CFG: + for what,cmd in (("load-only",[B,"load",p,"1"]),("load+vad33s",[CLI,"vad","--model",p,"--input",CLIP,"--threads","8"])): + r=subprocess.run(["/usr/bin/time","-v"]+cmd,capture_output=True,text=True) + m=re.search(r"Maximum resident set size \(kbytes\): (\d+)",r.stderr) + print(f"{n:22s} {what:12s} peak RSS {int(m.group(1))/1024:8.1f} MiB",flush=True) diff --git a/scripts/vad_bench/vad_slice_bench.cpp b/scripts/vad_bench/vad_slice_bench.cpp new file mode 100644 index 0000000..ec261c6 --- /dev/null +++ b/scripts/vad_bench/vad_slice_bench.cpp @@ -0,0 +1,44 @@ +// load N [cold] : N timed loads (ms each). speed : one timed VAD run after a warm-up. +#include +#include +#include +#include +#include +#include +#include +#include "model.hpp" +#include "silero_vad.hpp" +#include "vad_json.hpp" +#include "audio_io.hpp" +#include "ggml_graph.hpp" +using clk = std::chrono::steady_clock; +struct H { std::unique_ptr m; std::unique_ptr s; }; +static H load(const std::string& p) { + H h; std::string err; + if (pk::gguf_is_silero(p)) h.s = pk::SileroVad::load(p, &err); + else h.m = pk::gguf_is_vad_only(p) ? pk::Model::load_vad_only(p) : pk::Model::load(p); + if (!h.m && !h.s) { std::fprintf(stderr, "load failed\n"); std::exit(1); } + return h; +} +static std::string run(const H& h, const std::vector& pcm) { + pk::VadRequest r; r.kind = h.s ? pk::VadKind::kSilero : pk::VadKind::kHead; r.opts = pk::default_segmenter_opts(r.kind); + return h.s ? pk::silero_vad_to_json(*h.s, pcm, 16000, r) : pk::vad_to_json(*h.m, pcm, r); +} +int main(int argc, char** argv) { + std::string mode = argv[1], p = argv[2]; + if (mode == "load") { + int n = atoi(argv[3]); bool cold = argc > 4; + for (int i = 0; i < n; ++i) { + if (cold) { int fd = open(p.c_str(), O_RDONLY); posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED); close(fd); } + auto t0 = clk::now(); { H h = load(p); } + std::printf("%.1f\n", std::chrono::duration(clk::now() - t0).count()); + } + } else { + pk::Audio a; pk::load_audio_16k_mono(argv[3], a); + pk::set_num_threads(atoi(argv[4])); + H h = load(p); run(h, a.samples); + auto t0 = clk::now(); std::string o = run(h, a.samples); + double s = std::chrono::duration(clk::now() - t0).count(); + std::printf("%.4f %.1f\n", s, a.samples.size() / 16000.0 / s); + } +} From a6b03c925d083ef9c424a05f099fdf135bea1533 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 4 Oct 2026 06:56:22 +0000 Subject: [PATCH 4/4] docs: update the VAD benchmarks for the merged slice loader The VAD-only slice code (PR 87) is now in master. Say so, link the slice section of docs/vad.md instead of repeating it, and keep the note that no tagged release has it yet. Assisted-by: Claude:claude-sonnet-5-5 [Claude Code] --- docs/vad-benchmarks.md | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/docs/vad-benchmarks.md b/docs/vad-benchmarks.md index 693c6f6..ac02032 100644 --- a/docs/vad-benchmarks.md +++ b/docs/vad-benchmarks.md @@ -29,9 +29,9 @@ Accuracy is the frame level F1 (percent) against a coarse reference, see | Silero (matched settings) | 94.1 | 91.7 | 91.8 | about 660x on 1 thread, `parakeet-cli vad`, includes process start and model load | 60 MB (`parakeet-cli vad`, F32 and F16) | 2.2 MB (F32), 1.3 MB (F16) | | Silero (its own defaults) | 94.4 | 91.9 | 92.1 | not timed separately | same as above | same as above | | Ultra Q8_0 head, full ASR GGUF | 95.0 | 93.4 | 93.8 | 216x, 8 threads (older run); 91.6x on 1 thread and 266.6x on 8 threads in the slice run, busy machine | 905 MiB loaded, 1134 MiB after a 33 s clip | 941.5 MB | -| Ultra Q8_0 head, slice file (PR 87, not released) | same as the full file (see below) | same | same | 91.8x on 1 thread, 257.8x on 8 threads, busy machine | 12 MiB loaded, 241 MiB after a 33 s clip | 6.0 MB | +| Ultra Q8_0 head, slice file (PR 87, not in a release yet) | same as the full file (see below) | same | same | 91.8x on 1 thread, 257.8x on 8 threads, busy machine | 12 MiB loaded, 241 MiB after a 33 s clip | 6.0 MB | | Redux packed head, full ASR GGUF | 95.4 | 94.3 | 94.2 | 221x, 8 threads (older run); 94.9x on 1 thread and 267.1x on 8 threads in the slice run, busy machine | 372 MiB loaded, 603 MiB after a 33 s clip | 213.3 MB | -| Redux packed head, slice file (PR 87, not released) | same as the full file | same | same | 95.1x on 1 thread, 266.7x on 8 threads, busy machine | 16 MiB loaded, 246 MiB after a 33 s clip | 9.9 MB | +| Redux packed head, slice file (PR 87, not in a release yet) | same as the full file | same | same | 95.1x on 1 thread, 266.7x on 8 threads, busy machine | 16 MiB loaded, 246 MiB after a 33 s clip | 9.9 MB | | Silero F16, same build as the slice run | see Silero rows | | | 736x on 1 thread, 485x on 8 threads, busy machine | 9 MiB loaded, 13 MiB after a 33 s clip | 1.3 MB | | whisper.cpp Silero | 94.1 | not run | not run | in process: 686x on 1 thread | 36 MB (test harness) | 0.9 MB | | ONNX Silero (onnxruntime) | 94.0 | not run | not run | in process: 449x on 1 thread, Python loop | 104 MB | 2.3 MB | @@ -57,9 +57,11 @@ How to read it: - The speed columns of Silero and the heads are not a like for like comparison of the models. They show the cost of using each one as a stand alone gate. - The heads need the ASR GGUF unless you use a slice file. A slice is a GGUF with only - the head path, made by the code of PR 87 (open, not in a release). Its VAD output is + the head path, made by `scripts/slice_vad_gguf.py` (PR 87, in master, not in a tagged release yet). + Its VAD output is byte for byte the same as the parent file on three clips, so the accuracy columns - apply to the slice unchanged. See [Slice only head](#slice-only-head). + apply to the slice unchanged. See [Slice only head](#slice-only-head). The published + slices are listed in [vad.md](vad.md#published-slices). - All memory and load numbers for the heads are in the slice section. ### Machine and build @@ -442,8 +444,9 @@ and is not used above. ## Slice only head A "slice" is a GGUF that holds only the tensors the VAD path of an Ultra or Redux -file reads. The code that makes it and loads it is PR 87 -(`feat/vad-only-gguf`, open, not in a release). This section compares each full ASR +file reads. The code that makes it and loads it is PR 87 (merged to master as `e53a253`, not in a +tagged release yet; the latest tag is v0.5.0). How to make and use a slice is in +[vad.md](vad.md#vad-only-slice). This section compares each full ASR GGUF with its slice, and with Silero F16, built from the same source tree. ### Setup @@ -451,7 +454,7 @@ GGUF with its slice, and with Silero F16, built from the same source tree. - One build of the PR 87 code, one machine (see the table above), the first 300 s of the talk used elsewhere on this page for speed (`ted300`), and a 33 s clean synthetic clip for memory. The harness is `scripts/vad_bench/vad_slice_bench.cpp` (a small - program built inside the PR 87 tree, since it uses internal headers) and the driver is + program built inside the source tree of PR 87, since it uses internal headers) and the driver is `scripts/vad_bench/slice_bench.py`. - **Speed:** one timed VAD run after a warm-up run, 5 rounds with the 9 configurations interleaved, the best of the 5 reported (the median is in the result file), on 1 @@ -517,7 +520,7 @@ What this supports: 0.5 ms) are close to the timer's resolution and the maximum was up to 2.2 ms. - Peak memory was measured for a 33 s clip only. The peak memory for a 600 s clip was not measured, and the memory after the clip grows with the clip length. -- The slice code is in PR 87, which is open and not in a release. GPU backends and the +- The slice code is in master (PR 87) but not in a tagged release yet. GPU backends and the streaming VAD were not tested with slices. - These numbers come from one run of the harness. They were not repeated. - The same output check shows that the probabilities are identical, so the accuracy @@ -559,8 +562,8 @@ This is limited to what the numbers above support. 2 to 2.5 points higher on the TED talks, and 1.7 to 2.6 points higher at 0 dB pink noise. It has the higher recall and the lower precision. On these clips its start times are closer to the reference. -- **The Parakeet head as a stand alone detector.** With a slice file (PR 87, not in a - release) the head is a 6 to 10 MB file that loads in under 4 ms and needs about 240 +- **The Parakeet head as a stand alone detector.** With a slice file (PR 87, in master, not in a + tagged release) the head is a 6 to 10 MB file that loads in under 4 ms and needs about 240 to 250 MiB for a 33 s clip, with the same output as the full file. Without it, the head needs the full ASR GGUF (213 MB to 1.4 GB). Silero is still smaller (1.3 MB), uses less memory (13 MiB after the clip) and was about 8 times faster per core (736x against