diff --git a/README.md b/README.md index 05c1218..5621ca0 100644 --- a/README.md +++ b/README.md @@ -85,6 +85,8 @@ ternary ones. functions, as a stream (`parakeet_capi_vad_stream_*`), and as the cutter for any ASR model: `parakeet-cli transcribe --vad --vad-model silero.gguf`. See [`docs/vad.md`](docs/vad.md). - VAD-only slices of Ultra and Redux (6 to 10 MB, `redux-vad.gguf` and `ultra-vad-q8_0.gguf` in the same repo) hold just the head and its front end. They run `vad` and the `parakeet_capi_vad_*` calls and cannot transcribe. See [`docs/vad.md`](docs/vad.md). +- Accuracy, speed and size of both detectors, with the method and the scripts to repeat + them: [`docs/vad-benchmarks.md`](docs/vad-benchmarks.md). --- ## Performance diff --git a/docs/vad-benchmarks.md b/docs/vad-benchmarks.md new file mode 100644 index 0000000..ac02032 --- /dev/null +++ b/docs/vad-benchmarks.md @@ -0,0 +1,597 @@ +# VAD benchmarks + +This page lists every voice activity detection (VAD) measurement made for +parakeet.cpp so far: how each one was built, what it measures, the numbers, and +how to run it again. For the API and the model files, see [vad.md](vad.md). + +Read the caveats before you quote a number. Every reference here is coarse, the +machine was shared, and some runs were not on a quiet machine. A difference that +is smaller than the noise is not claimed. + +The detectors: + +| Name | What it is | +| --- | --- | +| Silero | Silero VAD 6.2.3 (MIT), 32 ms frames, from its own small GGUF (F32 or F16) | +| Ultra Q8_0 | VAD head of `moondream/parakeet-ultra`, GGUF in Q8_0, 80 ms frames | +| Redux packed | VAD head of `moondream/parakeet-redux`, packed ternary GGUF, 80 ms frames | +| whisper.cpp Silero | Silero VAD inside whisper.cpp (`whisper_vad_*`), used as a comparison | +| ONNX Silero | The official Silero ONNX file run with onnxruntime (CPU, one thread), used as the reference | + +## Summary + +Accuracy is the frame level F1 (percent) against a coarse reference, see +[Metrics](#metrics). Speed is audio seconds per wall second ("x real time") on a +300 s clip. Higher is faster. + +| Detector | F1, clean synthetic | F1, pink noise 0 dB | F1, TED-LIUM talks | Speed | Peak memory | File size | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| Silero (matched settings) | 94.1 | 91.7 | 91.8 | about 660x on 1 thread, `parakeet-cli vad`, includes process start and model load | 60 MB (`parakeet-cli vad`, F32 and F16) | 2.2 MB (F32), 1.3 MB (F16) | +| Silero (its own defaults) | 94.4 | 91.9 | 92.1 | not timed separately | same as above | same as above | +| Ultra Q8_0 head, full ASR GGUF | 95.0 | 93.4 | 93.8 | 216x, 8 threads (older run); 91.6x on 1 thread and 266.6x on 8 threads in the slice run, busy machine | 905 MiB loaded, 1134 MiB after a 33 s clip | 941.5 MB | +| Ultra Q8_0 head, slice file (PR 87, not in a release yet) | same as the full file (see below) | same | same | 91.8x on 1 thread, 257.8x on 8 threads, busy machine | 12 MiB loaded, 241 MiB after a 33 s clip | 6.0 MB | +| Redux packed head, full ASR GGUF | 95.4 | 94.3 | 94.2 | 221x, 8 threads (older run); 94.9x on 1 thread and 267.1x on 8 threads in the slice run, busy machine | 372 MiB loaded, 603 MiB after a 33 s clip | 213.3 MB | +| Redux packed head, slice file (PR 87, not in a release yet) | same as the full file | same | same | 95.1x on 1 thread, 266.7x on 8 threads, busy machine | 16 MiB loaded, 246 MiB after a 33 s clip | 9.9 MB | +| Silero F16, same build as the slice run | see Silero rows | | | 736x on 1 thread, 485x on 8 threads, busy machine | 9 MiB loaded, 13 MiB after a 33 s clip | 1.3 MB | +| whisper.cpp Silero | 94.1 | not run | not run | in process: 686x on 1 thread | 36 MB (test harness) | 0.9 MB | +| ONNX Silero (onnxruntime) | 94.0 | not run | not run | in process: 449x on 1 thread, Python loop | 104 MB | 2.3 MB | + +How to read it: + +- The synthetic and TED columns come from the + [heads against Silero](#parakeet-heads-against-silero) benchmark. The whisper.cpp + and ONNX rows come from the [Silero parity](#silero-in-whispercpp-and-in-parakeetcpp) + benchmark, which uses a different synthetic corpus (120 clips, 3 conditions). The + clean F1 of "Silero" and "whisper.cpp Silero" are therefore not from the same clips. + Compare rows only inside one benchmark. +- "Matched settings" means Silero ran with the same segmentation settings as the + heads (threshold 0.5, minimum speech 100 ms, minimum silence 200 ms, no padding). + See [Metrics](#metrics). +- Head speeds on 8 threads come from two separate runs on a busy machine (see the + caveats of each section): the older run (216x to 222x for four head builds, a Python + wrapper, load average about 6) and the slice run (255x to 268x, one build of the + C++ harness, load average up to 17 at the start of some rounds). They differ by about + 20%, which is the size of the noise on this machine for 8 threads, not a + difference between methods. The 1 thread rounds of the slice run were within about 5% of each other, + except for two outlier rounds (69.8x and 84.7x). +- The speed columns of Silero and the heads are not a like for like comparison of the + models. They show the cost of using each one as a stand alone gate. +- The heads need the ASR GGUF unless you use a slice file. A slice is a GGUF with only + the head path, made by `scripts/slice_vad_gguf.py` (PR 87, in master, not in a tagged release yet). + Its VAD output is + byte for byte the same as the parent file on three clips, so the accuracy columns + apply to the slice unchanged. See [Slice only head](#slice-only-head). The published + slices are listed in [vad.md](vad.md#published-slices). +- All memory and load numbers for the heads are in the slice section. + +### Machine and build + +| Item | Value | +| --- | --- | +| CPU | AMD Ryzen 9 class (Zen 5), 16-core desktop part; the OS shows 20 logical CPUs, one thread per core | +| Memory | 84 GB | +| OS and compiler | Linux 6.8, GCC 13.3, CMake 3.28 | +| ggml | commit e705c5fe (the submodule of this repository), `GGML_NATIVE=ON`, Release | +| parakeet.cpp | the VAD code of PR 83 (merged as `6165e3d`); see the per benchmark build notes below | +| whisper.cpp | commit `60c0be6ac8fa71b1a2ae2dd938a31a34a508e774` (version 1.9.4), CPU only | +| Load | the machine was shared with other jobs; load is given per benchmark | + +Build commits per benchmark: + +| Benchmark | parakeet.cpp build | +| --- | --- | +| Heads against Silero | a development build of the VAD branch; the commit was not recorded (unverified) | +| Silero in whisper.cpp and in parakeet.cpp | `c1a8391102d34074f48e758c441758327d2465b7`, the VAD branch before its squash into `6165e3d` | +| Long talks, batched decode | three builds, called base, b0 and b1: master before PR 81, PR 81 (`cbabaf5`), PR 82 (`9fdf3a1`). The result files do not record the commits; the mapping is from the PR 82 description (unverified) | +| Timing re-run | `6165e3d`, see [Timing re-run](#timing-re-run) | + +## Metrics + +All VAD output is reduced to speech regions (start and end in seconds). + +- **Frame level precision, recall, F1.** The regions of the detector and of the + reference are turned into a speech mask on a 10 ms grid. Counts of true positive, + false positive, false negative and true negative frames are summed over all clips + of a group, then precision, recall and F1 are computed from the sums. Pooled, not + an average of per clip scores. +- **Boundary error.** For each reference start (and end), the distance in ms to the + nearest detected start (end). Reported as median and 90th percentile, in ms, plus + the miss rate: the share of reference boundaries with no detected boundary within + 1 s. "All conditions" pools start and end errors. +- **Agreement.** Two detectors compared with each other: F1 of the two speech masks, + Cohen's kappa, and IoU (intersection over union) of the masks. +- **Probability parity.** Per frame, the absolute difference of the speech + probability against onnxruntime on the official ONNX file. Also the share of + frames where the decision at 0.5 flips. +- **Matched settings.** The segmentation step is not the same code in every system, + and the defaults differ. For a fair model comparison, Silero in the first + benchmark runs the `silero-vad` package (`get_speech_timestamps`) with threshold + 0.5, `min_speech_duration_ms` 100, `min_silence_duration_ms` 200, + `speech_pad_ms` 0. These are the head defaults. Silero's reference code also + lowers the threshold by 0.15 once speech has started (hysteresis); the parakeet.cpp + segmenter has no hysteresis (see [vad.md](vad.md)). "Silero (its own defaults)" uses + the package defaults (minimum speech 250 ms, minimum silence 100 ms, padding 30 ms). + The heads ran with their own defaults, which are the matched values. + +## Parakeet heads against Silero + +### Data + +Synthetic clips. Built from LibriSpeech test-clean (public). Every clip has five +utterances in a random order. Each utterance is trimmed at the first and last 10 ms +frame whose RMS is above 1% of the peak frame RMS. The utterances are joined with +gaps of 0.3 to 2.0 s (uniform), with 0.5 to 1.5 s before the first and after the +last. The gaps hold Gaussian noise at a standard deviation of 0.001. The reference +speech region of an utterance is the whole trimmed utterance, so short pauses inside +an utterance count as speech. + +Noise is added to the whole clip at a signal to noise ratio (SNR) measured against +the power of the speech regions: white or pink, at 20, 10, 5 and 0 dB, plus a clean +condition. That gives 9 conditions of 24 clips: **216 clips**, **1080 reference +regions**. The speech layout of clip k is the same in every condition. + +How the 24 clips were chosen is partly unverified. The script that cut the +LibriSpeech subset for this run was not saved. `scripts/vad_bench/fetch_data.py` +takes every 13th utterance of test-clean, which is the scheme of the Silero parity +corpus, and the numbers below were measured on a subset of that kind. A rebuild with +the script will give the same kind of corpus, not the same bytes, and the numbers +will differ slightly. + +TED-LIUM talks. Three talks from `distil-whisper/tedlium-long-form` (test split), +3178.6 s in total. Which three is not recorded in the result files (unverified). The +reference is built from word timestamps of the 0.6B v3 F16 ASR model run on 30 s +pieces: words closer than 0.4 s are merged into one region. This reference is +derived from an ASR model and is coarser than the synthetic one. It is also not +independent of the Parakeet family. The reference speech fraction is 91.9%. One more +talk was left out of the run by a name filter; the reason was not recorded. + +### Results: synthetic, threshold 0.5 + +Cells are precision / recall / F1, in percent. "Silero M" is Silero with matched +settings, "Silero D" with its own defaults. + +| Condition | Silero M | Silero D | Ultra Q8_0 | Redux packed | +| --- | --- | --- | --- | --- | +| clean | 99.1 / 89.7 / 94.1 | 98.7 / 90.5 / 94.4 | 98.2 / 92.0 / 95.0 | 98.4 / 92.7 / 95.4 | +| white 20 dB | 99.4 / 88.4 / 93.5 | 99.0 / 89.2 / 93.9 | 98.8 / 89.9 / 94.1 | 98.7 / 91.7 / 95.1 | +| white 10 dB | 99.3 / 88.0 / 93.3 | 98.9 / 89.0 / 93.7 | 98.9 / 88.3 / 93.3 | 98.2 / 91.9 / 94.9 | +| white 5 dB | 99.2 / 87.8 / 93.1 | 98.8 / 88.5 / 93.4 | 98.9 / 87.2 / 92.7 | 97.4 / 91.9 / 94.6 | +| white 0 dB | 99.2 / 85.8 / 92.0 | 98.8 / 86.7 / 92.4 | 98.3 / 87.9 / 92.8 | 96.4 / 92.3 / 94.3 | +| pink 20 dB | 99.4 / 88.4 / 93.6 | 99.1 / 89.3 / 93.9 | 98.8 / 90.8 / 94.6 | 98.6 / 92.0 / 95.2 | +| pink 10 dB | 99.3 / 88.0 / 93.3 | 99.0 / 89.0 / 93.7 | 99.0 / 88.8 / 93.6 | 98.1 / 92.2 / 95.1 | +| pink 5 dB | 99.4 / 87.6 / 93.1 | 99.0 / 88.5 / 93.5 | 99.0 / 88.4 / 93.4 | 97.4 / 92.6 / 94.9 | +| pink 0 dB | 99.2 / 85.2 / 91.7 | 98.8 / 85.9 / 91.9 | 96.3 / 90.8 / 93.4 | 95.5 / 93.2 / 94.3 | + +What the table supports: the three detectors are within about 1.5 F1 points of each +other at 5 dB and above, and up to 2.6 points apart at 0 dB. Silero has the higher precision and the lower recall. Both +heads have the higher recall. In the loudest noise (0 dB) the heads keep their F1 +better than Silero (pink 0 dB: Silero M 91.7, Ultra 93.4, Redux 94.3). The recall +of all detectors is below 94% because the reference counts pauses inside utterances +as speech, so the absolute recall values are not a measure of missed speech. + +Boundary error, in ms (median / 90th percentile, miss rate), start then end: + +| Condition | Silero M | Silero D | Ultra Q8_0 | Redux packed | +| --- | --- | --- | --- | --- | +| clean | S 171/527 (0.0%) E 121/366 (0.0%) | S 162/499 (0.0%) E 137/343 (0.0%) | S 123/460 (0.0%) E 168/325 (0.0%) | S 152/459 (0.0%) E 151/328 (0.0%) | +| white 10 dB | S 249/526 (0.0%) E 116/371 (0.0%) | S 232/500 (0.0%) E 129/357 (0.0%) | S 211/470 (0.0%) E 147/472 (0.8%) | S 163/466 (0.8%) E 152/389 (0.8%) | +| white 0 dB | S 262/571 (0.8%) E 117/459 (2.5%) | S 237/526 (0.8%) E 142/451 (1.7%) | S 162/461 (0.0%) E 149/494 (2.5%) | S 202/460 (0.8%) E 149/404 (1.7%) | +| pink 10 dB | S 248/526 (0.0%) E 106/384 (0.0%) | S 232/502 (0.0%) E 117/343 (0.0%) | S 226/479 (0.0%) E 136/445 (0.0%) | S 176/445 (0.0%) E 148/340 (0.0%) | +| pink 0 dB | S 261/570 (0.0%) E 145/411 (3.3%) | S 248/530 (0.0%) E 150/408 (3.3%) | S 180/457 (2.5%) E 162/470 (2.5%) | S 200/457 (1.7%) E 160/428 (1.7%) | + +All conditions pooled: + +| System | Median | 90th percentile | Start median | End median | Miss rate | +| --- | ---: | ---: | ---: | ---: | ---: | +| Silero M | 143 | 501 | 237 | 116 | 0.4% | +| Silero D | 149 | 484 | 232 | 130 | 0.3% | +| Ultra Q8_0 | 154 | 465 | 181 | 149 | 0.6% | +| Redux packed | 153 | 437 | 170 | 149 | 0.4% | + +Boundary errors of 100 to 250 ms are large next to the 32 and 80 ms frame sizes. +Most of this comes from the reference (trimmed utterances next to short gaps), so +only the ordering of start errors is informative: the heads place starts about 50 +to 70 ms closer to the reference than Silero M does. The signed start error median +is 237 ms for Silero M, 232 for Silero D, 162 for Ultra and 147 for Redux (the +detector starts later than the reference). The signed end error median is 3, 20, -10 +and 54 ms. Segments per reference region: Silero M 1.96, Ultra 1.92, Redux 1.76 +(a value above 1 means the detector splits utterances at their inner pauses). + +Threshold sweep, pooled over clean, white 10 dB, white 0 dB and pink 10 dB +(precision / recall / F1): + +| System | threshold 0.3 | threshold 0.5 | threshold 0.7 | +| --- | --- | --- | --- | +| Silero M | 98.5 / 90.3 / 94.3 | 99.2 / 87.9 / 93.2 | 99.5 / 86.2 / 92.4 | +| Ultra Q8_0 | 97.1 / 92.4 / 94.7 | 98.6 / 89.3 / 93.7 | 99.0 / 86.2 / 92.1 | +| Redux packed | 94.8 / 95.0 / 94.9 | 97.7 / 92.3 / 94.9 | 98.9 / 90.0 / 94.2 | + +Agreement with Silero M on all synthetic clips: Ultra F1 96.6, kappa 0.882, IoU 93.4. +Redux F1 96.3, kappa 0.865, IoU 92.8. Ultra against Redux: F1 97.2, kappa 0.896. + +### Results: TED-LIUM talks + +Frame level on 3178.6 s, with the ASR derived reference (speech fraction 91.9%): + +| System | Precision | Recall | F1 | Speech fraction found | +| --- | ---: | ---: | ---: | ---: | +| Silero M | 97.3 | 86.8 | 91.8 | 82.0% | +| Silero D | 97.4 | 87.4 | 92.1 | not recorded | +| Ultra Q8_0 | 97.1 | 90.7 | 93.8 | 85.8% | +| Redux packed | 97.1 | 91.5 | 94.2 | 86.6% | + +Agreement: Ultra and Silero M F1 97.0, kappa 0.813; Redux and Silero M F1 96.9, +kappa 0.801; Ultra and Redux F1 98.8, kappa 0.912. The "Silero D" speech fraction +was not printed in the run output. + +The ordering is the same as on the synthetic clips: equal precision, the heads +have the higher recall. The gap in F1 between Silero M and the heads is 2 to 2.5 +points. Because the reference comes from a Parakeet model, a bias in favor of the +Parakeet heads is possible and was not measured. + +### Results: speed of the heads and of Silero ONNX + +300 s of one talk (the first 300 s of the 1249 s talk), 4 repetitions, load average +5.8 to 6.2 during the run. Silero ran through the `silero-vad` Python package +(ONNX, one thread). The heads ran with 8 threads. + +| System | Speed, x real time | +| --- | ---: | +| Silero ONNX, 1 thread | 393 | +| Ultra Q8_0, 8 threads | 216 | +| Ultra F16, 8 threads | 218 | +| Redux packed, 8 threads | 221 | +| Redux dequantized F16, 8 threads | 222 | + +The run printed one speed per system. The result file does not say whether it is the +best or the median of the 4 repetitions (unverified). The speeds that the TED script +printed during the accuracy run (load average 6 to 42) are not valid and are not +used. Thread pinning was not recorded for this run. + +### Caveats + +- References are coarse. The synthetic reference counts pauses inside utterances as + speech and uses an energy trim. The TED reference comes from ASR word times. +- Absolute precision and recall depend on that reference. Compare detectors with + each other, on the same clips. +- Differences of less than about one F1 point between two detectors in one cell are + not claimed. There are no confidence intervals; one corpus was used. +- Speed numbers: see the load notes above. Detectors with different thread counts + are not a model comparison. +- The build commit of this run was not recorded. + +## Silero in whisper.cpp and in parakeet.cpp + +This benchmark asks whether the Silero port in parakeet.cpp gives the same result +and the same speed as the Silero VAD that ships in whisper.cpp. + +### Setup + +- **Models.** `silero_vad.onnx` (version 6.2.3, SHA-256 + `1a153a22f4509e292a94e67d6f9b85e8deb25b4988682b7e174c65279d8788e3`); + parakeet.cpp GGUF F32 (`1398e5ce...`) and F16 (`8160489282...`) made with + `scripts/convert_silero_vad_to_gguf.py` from that ONNX file; whisper.cpp model + `ggml-silero-v6.2.3-ggml.bin` (`abd79739...`), converted from the same ONNX with the + whisper.cpp conversion script, and the model of the whisper.cpp model repository + `ggml-silero-v6.2.0.bin` (`2aa269b7...`). The full list of hashes and versions is in + [`silero_versions_and_sha256.txt`](../scripts/vad_bench/results/silero_versions_and_sha256.txt). +- **Reference.** onnxruntime on the official ONNX file, with the official 64 sample + context, one thread. +- **whisper.cpp access.** A small test program (`scripts/vad_bench/wvad.cpp`) over the + public `whisper_vad_*` functions: probabilities, and the segments of the whisper.cpp + C++ code with its defaults and with matched parameters. It is not part of + whisper.cpp. +- **Corpus.** 120 synthetic clips (40 clips of 5 utterances, in clean, white 10 dB and + pink 10 dB), built with the same recipe as above from 200 LibriSpeech test-clean + utterances (every 13th). Plus the first 600 s of one TED-LIUM talk. +- **Segmentation.** To compare only the models, the same Python code turns every + probability stream into segments: threshold 0.5, negative threshold 0.35, minimum + speech 100 ms, minimum silence 200 ms, padding 0 ("matched post"). The own defaults + of whisper.cpp and parakeet.cpp are also shown. + +### Probability parity against onnxruntime + +All 120 clips and the 600 s talk, one value per 32 ms chunk: + +| System | Max abs diff | Mean abs diff | Frames > 0.05 off | Decision agreement at 0.5 | Flipped frames | +| --- | ---: | ---: | ---: | ---: | ---: | +| whisper.cpp, converted 6.2.3 | 0.7373 | 0.00224 | 0.54% | 99.826% | 309 | +| whisper.cpp, repository 6.2.0 | 0.7373 | 0.00224 | 0.54% | 99.826% | 309 | +| parakeet.cpp F32 | 0.0001 | 0.00002 | 0.00% | 99.999% | 2 | +| parakeet.cpp F16 | 0.0036 | 0.00005 | 0.00% | 99.997% | 5 | + +whisper.cpp (converted 6.2.3) against parakeet.cpp F32: mean abs diff 0.00224, 311 +flipped frames. The two whisper.cpp model files give the same result (0 flipped +frames between them). Mean abs diff against onnxruntime, first chunk of each clip / +the rest: whisper.cpp 0.00088 / 0.00212, parakeet.cpp F32 0.00002 / 0.00002, F16 +0.00005 / 0.00005. On the 600 s talk alone, the number of flipped frames against +onnxruntime is 57 of 18750 for whisper.cpp, 0 of 18750 for parakeet.cpp F32 and 1 +for F16. The cause of the whisper.cpp difference was not investigated. + +### Segment quality on the synthetic clips + +Precision / recall / F1 in percent, 40 clips per condition, same reference as above: + +| System (settings) | clean | white 10 dB | pink 10 dB | all | +| --- | --- | --- | --- | --- | +| onnxruntime (matched post) | 99.3 / 89.3 / 94.0 | 99.4 / 87.8 / 93.2 | 99.5 / 87.4 / 93.1 | 99.4 / 88.2 / 93.5 | +| whisper.cpp (matched post) | 99.3 / 89.4 / 94.1 | 99.4 / 87.8 / 93.3 | 99.5 / 87.4 / 93.0 | 99.4 / 88.2 / 93.5 | +| parakeet.cpp F32 (matched post) | 99.3 / 89.3 / 94.0 | 99.4 / 87.8 / 93.2 | 99.5 / 87.4 / 93.1 | 99.4 / 88.2 / 93.5 | +| parakeet.cpp F16 (matched post) | 99.3 / 89.3 / 94.0 | 99.4 / 87.8 / 93.2 | 99.5 / 87.4 / 93.1 | 99.4 / 88.2 / 93.5 | +| whisper.cpp (its own defaults) | 98.9 / 90.3 / 94.4 | 99.2 / 89.0 / 93.8 | 99.3 / 88.6 / 93.6 | 99.1 / 89.3 / 94.0 | +| whisper.cpp (its C++ code, matched parameters) | 99.3 / 89.4 / 94.1 | 99.4 / 87.8 / 93.3 | 99.5 / 87.4 / 93.0 | 99.4 / 88.2 / 93.5 | +| parakeet.cpp F32 (its own defaults) | 99.1 / 89.3 / 93.9 | 99.3 / 87.6 / 93.1 | 99.4 / 87.3 / 93.0 | 99.3 / 88.1 / 93.3 | + +Boundary error against the reference, all conditions pooled (start median / 90th +percentile, end median / 90th percentile, miss rate): onnxruntime, whisper.cpp, +parakeet.cpp F32 and F16 with matched post all give 241/553 and 108/396, 0.2%. +whisper.cpp with its own defaults gives 209/521, 124/367, 0.0%. parakeet.cpp with its +own defaults gives 211/523, 116/392, 0.0%. + +Agreement of the speech masks (matched post), 120 clips pooled (F1 / kappa / IoU): + +| Pair | F1 | kappa | IoU | +| --- | ---: | ---: | ---: | +| whisper.cpp vs parakeet.cpp F32 | 99.92 | 0.9969 | 99.84 | +| whisper.cpp vs onnxruntime | 99.92 | 0.9969 | 99.84 | +| parakeet.cpp F32 vs onnxruntime | 100.00 | 1.0000 | 100.00 | +| parakeet.cpp F16 vs F32 | 100.00 | 0.9999 | 100.00 | +| whisper.cpp 6.2.3 vs 6.2.0 file | 100.00 | 1.0000 | 100.00 | + +On the 600 s talk (no reference), speech fraction found: onnxruntime 79.1%, +whisper.cpp 78.8%, parakeet.cpp F32 79.1%. whisper.cpp against parakeet.cpp F32: +F1 99.75, kappa 0.9882. + +What this supports: with the same post processing, whisper.cpp Silero and parakeet.cpp +Silero give the same segment quality on these clips (all within 0.1 F1 point). The +difference between whisper.cpp and parakeet.cpp in their own defaults (94.0 against +93.3 F1 on all clips) comes from their different default settings, not from the model. +The F16 file is not measurably worse than F32 here. + +### Speed + +Each job ran on one pinned core, on a 300 s clip (the first 300 s of the 600 s talk; the +clip file was not kept, so this is assumed from the preparation script), +serialized with a lock. A run started only when the mean number of other runnable +tasks over 10 s was below 4, or the 1-minute load average was below 5. The machine has +20 logical CPUs and was never idle, so the 1-minute load average at the start of the +runs was between 4.2 and 15.9, and the mean runnable count was 1.8 to 5.5. These are +not quiet machine numbers. Wall time is the process wall time, including +process start and model load for the command line tools. + +| Job | Runs | Wall time, median (min to max), s | In process time, median, s | x real time (median) | Peak RSS | +| --- | ---: | ---: | ---: | ---: | ---: | +| whisper.cpp `detect_speech` only (test program) | 5 | 0.456 (0.446 to 0.473) | 0.438 | 686 | 36 MB | +| `whisper-vad-speech-segments` | 5 | 0.452 (0.444 to 0.457) | not applicable | 664 | 28 MB | +| `parakeet-cli vad`, F32 | 5 | 0.455 (0.425 to 0.464) | not applicable | 659 | 60 MB | +| `parakeet-cli vad`, F16 | 5 | 0.451 (0.436 to 0.553) | not applicable | 666 | 60 MB | +| onnxruntime, Python, 1 thread | 3 | 0.806 (0.738 to 1.241) | 0.668 | 449 | 104 MB | + +The x real time column is 300 s divided by the in process time when there is one, +else by the wall time. The four Silero command line jobs are within the spread of +their repeated runs (0.43 to 0.55 s), so the numbers do not show a speed difference +between whisper.cpp and parakeet.cpp. The onnxruntime figure is a Python loop that +calls the session once per 512 sample chunk. It is not a tuned onnxruntime number. + +whisper.cpp with `tiny.en`, one thread, same 300 s clip, end to end: 15.14 s +without VAD, 19.68 s with the Silero VAD (3 runs each, load average 4.6 to 8.3). The +ASR part dominates. Why the run with VAD is slower was not investigated. + +### Caveats + +- The load notes above apply. The runs were on a shared machine. +- One clip and one core. No claim is made about other lengths, other CPUs, or more + threads. +- The onnxruntime run had 3 repetitions. The other jobs had 5. +- The measured whisper.cpp is a source build at the commit above, CPU only. + +## Long talks: batched decode for VAD segments + +PR 82 decodes the VAD segments in groups of up to 16, using the exact batched +decoder of PR 81. Each segment is still encoded alone. This measures that change, not +the VAD itself. + +Data: four TED-LIUM long-form talks of 18 to 25 minutes, merged to one file each. +Models: Ultra Q8_0 and Redux packed. Command: `parakeet-cli transcribe --vad --json +--threads 8`, pinned to 8 cores. Three builds, interleaved, 5 runs each: +**120 runs**. The build order was rotated for each repetition. Before each run the +script waited until the mean number of other runnable tasks over 10 s was below 4. +The mean number of other runnable tasks at the gate was 0 to 1.5, but the 1-minute +load average at the start of the runs was 2.5 to 18.0 (it includes the earlier runs of +the same job and other jobs), see +`scripts/vad_bench/results/longform_b1_loadlog.txt` and the per run values in +`longform_b1_results.tsv`. The 1-minute load average was therefore often above 5. +Wall time is the time of the whole command, including model load. + +Median wall time in seconds over 5 runs: + +| Talk (length, s) | Model | Segments | base | PR 81 | PR 82 | PR 82 vs PR 81 | Peak RSS PR 82 vs PR 81 | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| BillGates (1505.8) | Ultra Q8_0 | 59 | 44.2 | 43.7 | 40.8 | 1.072x | +1.7% | +| BillGates | Redux packed | 62 | 28.1 | 27.7 | 25.9 | 1.071x | +2.2% | +| AimeeMullins (1249.0) | Ultra Q8_0 | 46 | 36.9 | 36.9 | 34.8 | 1.058x | +1.7% | +| AimeeMullins | Redux packed | 46 | 23.4 | 23.7 | 21.9 | 1.079x | +2.5% | +| JaneMcGonigal (1167.6) | Ultra Q8_0 | 50 | 35.9 | 34.9 | 34.0 | 1.029x | +1.4% | +| JaneMcGonigal | Redux packed | 47 | 24.1 | 23.1 | 23.0 | 1.006x | +2.2% | +| DanielKahneman (1095.5) | Ultra Q8_0 | 44 | 32.6 | 33.1 | 32.9 | 1.006x | +1.7% | +| DanielKahneman | Redux packed | 42 | 21.4 | 21.5 | 21.2 | 1.013x | +2.2% | + +The average of the eight speed ratios is 1.042x (range 1.006x to 1.079x). Peak +memory of the whole transcription is 1.4 to 2.5 percent higher with PR 82, 30 to 40 +MB on a base of 1.4 to 2.0 GB. The transcript (JSON output) has the same hash for all +three builds in all 8 talk and model pairs. + +Caveats: the gain is small and several of the ratios are inside the spread of the +5 runs (some single runs of the same configuration differ by more than 3%); do not +claim more than a small gain. Only CPU and x86 were measured. The memory numbers are +for the whole transcription, not for the VAD. + +## Silero parity with the reference (PR 83) + +This table is also in [vad.md](vad.md). Test clip: `tests/fixtures/speech.wav` padded +with silence (310 chunks), against onnxruntime 1.27.0 and the ONNX file of version +6.2.3. Maximum probability difference: + +| File | 16 kHz | 8 kHz | +| --- | ---: | ---: | +| F32 | 9e-7 | 2e-6 | +| F16 | 7e-4 | 3e-3 | + +One run of `tests/silero_vad_probe` on one CPU thread measured about 40 us per chunk +for F32 and F16 at both rates. That is one run on one machine. It is not a benchmark +and is not used above. + +## Slice only head + +A "slice" is a GGUF that holds only the tensors the VAD path of an Ultra or Redux +file reads. The code that makes it and loads it is PR 87 (merged to master as `e53a253`, not in a +tagged release yet; the latest tag is v0.5.0). How to make and use a slice is in +[vad.md](vad.md#vad-only-slice). This section compares each full ASR +GGUF with its slice, and with Silero F16, built from the same source tree. + +### Setup + +- One build of the PR 87 code, one machine (see the table above), the first 300 s of + the talk used elsewhere on this page for speed (`ted300`), and a 33 s clean synthetic + clip for memory. The harness is `scripts/vad_bench/vad_slice_bench.cpp` (a small + program built inside the source tree of PR 87, since it uses internal headers) and the driver is + `scripts/vad_bench/slice_bench.py`. +- **Speed:** one timed VAD run after a warm-up run, 5 rounds with the 9 configurations + interleaved, the best of the 5 reported (the median is in the result file), on 1 + thread and on 8 threads. The 8 thread runs were pinned to the 8 idlest cores at the time. + Each round waited for the gate: mean number of other runnable tasks under 4 or the + 1-minute load average under 5. +- **Load time:** the model load alone, median of 10. Warm means the file is in the + page cache (the first load, which warms it, is dropped). Cold means the page cache + for the file was evicted before each load. +- **Peak memory:** peak resident set size from `/usr/bin/time -v`, once per + configuration: loading only, and `parakeet-cli vad` on the 33 s clip with 8 threads. +- **Same output:** `parakeet-cli vad --probabilities` was run with each slice and + with its parent file on three clips (the 33 s clip, a noisy clip, the 600 s talk). + All four slices gave byte-identical JSON on all three clips (12 of 12 pairs). + +### Results + +| Model | Full file | Slice file | Load, warm / cold, full (ms) | Load, warm / cold, slice (ms) | Peak RSS loaded, full / slice (MiB) | Peak RSS after the 33 s clip, full / slice (MiB) | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| Ultra Q8_0 | 941.5 MB | 6.0 MB | 276 / 424 | 0.2 / 2.1 | 905 / 12 | 1134 / 241 | +| Ultra F16 | 1441.9 MB | 9.9 MB | 429 / 666 | 0.4 / 3.5 | 1382 / 15 | 1613 / 246 | +| Redux packed | 213.3 MB | 9.9 MB | 92 / 128 | 0.5 / 3.7 | 372 / 16 | 603 / 246 | +| Redux dequantized F16 | 1441.9 MB | 9.9 MB | 434 / 669 | 0.2 / 3.8 | 1382 / 16 | 1613 / 246 | +| Silero F16 | 1.3 MB | not applicable | 0.4 / 1.5 | not applicable | 9 / not applicable | 13 / not applicable | + +Speed on the 300 s clip, x real time, best of 5 (1 thread / 8 threads): + +| Model | Full file | Slice file | +| --- | ---: | ---: | +| Ultra Q8_0 | 91.6 / 266.6 | 91.8 / 257.8 | +| Ultra F16 | 94.5 / 255.5 | 94.6 / 265.3 | +| Redux packed | 94.9 / 267.1 | 95.1 / 266.7 | +| Redux dequantized F16 | 95.1 / 265.3 | 94.8 / 268.1 | +| Silero F16 | 736.3 / 485.4 | not applicable | + +Medians of the 5 rounds (1 thread / 8 threads) are lower: Ultra Q8_0 full 91.4 / 250.5, +slice 91.5 / 227.8; Redux packed full 93.1 / 252.8, slice 92.4 / 236.3; Silero F16 +702.1 / 473.9. The slice files are 6,007,328 bytes (Ultra Q8_0) and 9,939,488 bytes (the other +three). The SHA-256 of the four slice files is in +[`slice_sha256.txt`](../scripts/vad_bench/results/slice_sha256.txt). + +What this supports: + +- A slice is 6 to 10 MB, loads in under 4 ms, and needs about 12 to 16 MiB loaded and + about 240 to 250 MiB for a 33 s clip. The full file needs 372 MiB to 1.4 GiB just to + load. +- Slice and full file run at the same speed. The differences between them on 1 thread (best of 5) are + 0.1% to 0.3%, far inside the spread of the 5 rounds. +- Silero F16 is smaller (1.3 MB, 13 MiB after the clip), loads in about 0.4 ms, and was + about 7.7 to 8 times faster than the heads on 1 thread (736x against 92x to 95x). + +### Caveats + +- The 8 thread numbers are noisy. The machine was busy: the gate lines of the speed + rounds in `slice_gate.log` show a 1-minute load average from 4.2 to 17.3 at the start + of the rounds, and some rounds ran at 9 to 17 (the gate also accepts a low runnable + count when the load average is high). The best of 5 on 8 threads is within the noise + of the other values (the 8 thread rounds of one configuration ranged by up to 30%, for + example Ultra Q8_0 full from 185.0x to 266.6x). Do not read a ranking from the 8 thread + column. The 1 thread rounds of one configuration are within about 5% of each other, + except for two outlier rounds (Redux dequantized F16 full 69.8x, slice 84.7x). +- Load times are medians of 10 and tiny for slices, so the warm slice values (0.2 to + 0.5 ms) are close to the timer's resolution and the maximum was up to 2.2 ms. +- Peak memory was measured for a 33 s clip only. The peak memory for a 600 s clip was + not measured, and the memory after the clip grows with the clip length. +- The slice code is in master (PR 87) but not in a tagged release yet. GPU backends and the + streaming VAD were not tested with slices. +- These numbers come from one run of the harness. They were not repeated. +- The same output check shows that the probabilities are identical, so the accuracy + tables of this page apply to the slices. It was run on three clips, not on the whole + corpus. + +## Timing re-run + +A re-run of the timing parts on a quiet machine was tried and did not happen. The +rule was: run only when the 1-minute load average is below 5, check before each run, +poll every 10 minutes for up to 3 hours, serialize with a lock and pin cores. +The machine was shared with other jobs. The load average was checked 19 times, from +22:43 to 01:43, and was never below 5. The lowest value was 6.67 and the highest was +82.37. The poll log is in +[`timing_rerun_load_log.txt`](../scripts/vad_bench/results/timing_rerun_load_log.txt). +No timing was run for this page. The speed numbers of the slice section were taken +separately, on a busy machine, as described there. They are not quiet-machine numbers +either. The speed numbers on this +page are the old ones, taken under the loads that each section states. Numbers taken +on a busy machine: all speeds of the heads against Silero section, all Silero speed +and memory numbers of the parity section, all slice section numbers, and the batched +decode times (load average 2.5 to 18.0 at the start). The setup for the re-run (a build of `6165e3d`, the 300 s clip, the +models and the driver) was ready, and `speed_silero.py` and `speed_heads.py` run it in +one command each when the machine is quiet. + +The accuracy numbers do not depend on load and were not re-run. + +## When to use which + +This is limited to what the numbers above support. + +- **Stand alone gate, any ASR model, small files.** Silero. Its GGUF is 1.3 to 2.2 + MB, the command line tool needs about 60 MB, and it ran at about 660x real time on + one thread on a shared machine. It gave frame level F1 of 91.7 to 94.4 on the + synthetic data and 91.8 to 92.1 on the TED talks. +- **The Parakeet head when Ultra or Redux is already loaded.** The head costs no extra + model and its scores are close to Silero in the same tests: within about 1.5 F1 points + on the synthetic clips at 5 dB and above (Ultra was 0.4 lower in white noise at 5 dB), + 2 to 2.5 points higher on the TED talks, and 1.7 to 2.6 points higher at 0 dB pink + noise. It has the higher recall and the lower precision. On these clips its start + times are closer to the reference. +- **The Parakeet head as a stand alone detector.** With a slice file (PR 87, in master, not in a + tagged release) the head is a 6 to 10 MB file that loads in under 4 ms and needs about 240 + to 250 MiB for a 33 s clip, with the same output as the full file. Without it, the head + needs the full ASR GGUF (213 MB to 1.4 GB). Silero is still smaller (1.3 MB), uses less + memory (13 MiB after the clip) and was about 8 times faster per core (736x against + 92x to 95x on 1 thread, on a busy machine). Choose the head for its accuracy on the + tests above, Silero for size and speed. +- **Silero from whisper.cpp or from parakeet.cpp.** On the 120 clips and the 600 s talk + the two give the same segments (mask F1 99.92) and the same speed within the noise of + a shared machine. Choose by the rest of your stack, not by quality or speed. The + probabilities of parakeet.cpp match onnxruntime more closely (mean abs diff 0.00002, + 2 flipped frames) than those of whisper.cpp do (0.00224, 309 flipped frames), but this + did not change the segment scores. +- **Not supported by these numbers:** any claim about GPUs, ARM, other languages, + music or non speech noise, streaming latency, or the effect of the VAD on WER. + +## How to reproduce + +The scripts are in [`scripts/vad_bench/`](../scripts/vad_bench/), with a README that +lists the inputs, the commands and the Python packages. No audio and no model files +are committed. In short: + +1. Fetch the public audio: `fetch_data.py`, `fetch_ted_talks.py`. +2. Slice section: `vad_slice_bench.cpp` (build it inside the PR 87 tree) and `slice_bench.py`. +3. Heads against Silero: `compare_synthetic.py` then `analyze_synthetic.py`; + `ted_reference.py` then `compare_ted.py`; `speed_heads.py`. +4. Silero parity: `make_clips.py`, build `wvad.cpp` against whisper.cpp, + `silero_collect.py`, `silero_analyze.py`; speed with `speed_silero.py`. +5. Long talks: `longform_b1.sh`. + +Small result files of the runs on this page (tables, per run timings and load logs) +are in `scripts/vad_bench/results/`. The raw per clip predictions are not committed; +a re-run regenerates them. diff --git a/docs/vad.md b/docs/vad.md index 0155674..a8eda25 100644 --- a/docs/vad.md +++ b/docs/vad.md @@ -12,6 +12,9 @@ Both give one speech probability per frame. A shared segmenter transcription segments. The segmenter takes the frame period as a parameter (`SegmenterOpts::frame_sec`); only the option defaults differ per detector. +Accuracy and speed numbers for both detectors, with the method and the scripts to +repeat them, are in [vad-benchmarks.md](vad-benchmarks.md). + ## Standalone VAD API One call returns the speech regions of a clip as JSON, for either detector. Load @@ -242,7 +245,7 @@ onnxruntime 1.27.0 and the ONNX file of version 6.2.3: On one CPU thread, `tests/silero_vad_probe` measured about 40 us per chunk (F32 or F16, 16 kHz or 8 kHz) on the development machine. That is one run on one -machine, not a benchmark. +machine, not a benchmark. For benchmarks, see [vad-benchmarks.md](vad-benchmarks.md). ## Tests diff --git a/scripts/vad_bench/README.md b/scripts/vad_bench/README.md new file mode 100644 index 0000000..7deb9fc --- /dev/null +++ b/scripts/vad_bench/README.md @@ -0,0 +1,75 @@ +# VAD benchmark scripts + +Scripts behind [docs/vad-benchmarks.md](../../docs/vad-benchmarks.md). The page says what +each number means and which caveats apply. No audio and no model files are committed; +`results/` holds the small result files of the runs on that page. + +All paths are arguments. Run from the repository root. `PARAKEET_LIB` is the path of +`libparakeet.so` (build with `-DPARAKEET_SHARED=ON`; the default is `build/libparakeet.so`). + +Python packages: `numpy soundfile librosa datasets torch onnxruntime silero-vad` +(the Silero reference is `silero-vad` 6.2.3). Models: the ASR GGUFs of +`moondream/parakeet-ultra` (Q8_0 and F16), `moondream/parakeet-redux` (packed `keep` and +dequantized F16), a TDT 0.6B v3 F16 GGUF for the reference of the TED talks, and the +Silero GGUFs made with `scripts/convert_silero_vad_to_gguf.py`. + +## Data + +| Script | What | +| --- | --- | +| `fetch_data.py OUT [--libri-count N] [--ted-seconds S]` | LibriSpeech test-clean (every 13th utterance) as `libri.npy`, and the first 600 s of one TED-LIUM talk | +| `fetch_ted_talks.py OUT [N]` | the first N TED-LIUM long-form talks under 1500 s as `ted_.wav` | +| `make_clips.py DATA` | the 120 synthetic clips (clean, white 10 dB, pink 10 dB) of the Silero parity run | +| `vad_synth.py` | clip builder, noise, and metrics (frame F1, kappa, boundary error) | + +## Parakeet heads against Silero + +``` +scripts/vad_bench/compare_synthetic.py DATA GGUF_DIR out/synthetic_raw.json 24 +scripts/vad_bench/analyze_synthetic.py out/synthetic_raw.json +scripts/vad_bench/ted_reference.py DATA tdt-0.6b-v3-f16.gguf # words of each talk +scripts/vad_bench/compare_ted.py DATA GGUF_DIR out/ted_metrics.json +taskset -c 0-7 scripts/vad_bench/speed_heads.py talk.wav GGUF_DIR # timing, quiet machine only +``` + +`GGUF_DIR` holds `ultra-q8_0.gguf`, `ultra-f16.gguf`, `redux-keep.gguf` and +`redux-deq-f16.gguf`. `compare_ted.py` prints speeds; ignore them, use `speed_heads.py`. + +## Silero in whisper.cpp and in parakeet.cpp + +Build whisper.cpp (CPU), then the test program, which uses its public `whisper_vad_*` API: + +``` +g++ -O2 -std=c++17 scripts/vad_bench/wvad.cpp -I whisper.cpp/include -I whisper.cpp/ggml/include \ + -L whisper.cpp/build/bin -lwhisper -Wl,-rpath,$PWD/whisper.cpp/build/bin -o wvad +python3 scripts/vad_bench/silero_collect.py CORE DATA MODEL_DIR build/examples/cli/parakeet-cli ./wvad +python3 scripts/vad_bench/silero_analyze.py DATA # run where probs.npz was written +python3 scripts/vad_bench/speed_silero.py CORE ROUNDS DATA/ted300.wav MODEL_DIR \ + build/examples/cli/parakeet-cli ./wvad whisper.cpp/build/bin out/speed.json +``` + +`ted300.wav` is the first 300 s of `ted_talk.wav`. `MODEL_DIR` holds `silero_vad.onnx`, +`silero-vad-f32.gguf`, `silero-vad-f16.gguf`, `ggml-silero-v6.2.3-ggml.bin`, +`ggml-silero-v6.2.0.bin` and, for the end to end job, `ggml-tiny.en.bin`. +`silero_analyze.py` expects `probs.npz` and `segs_own_default.json` in the current directory. + +## Long talks, batched decode + +``` +scripts/vad_bench/longform_b1.sh TALKS_DIR GGUF_DIR BUILDS_DIR out.tsv loadlog.txt +``` + +`BUILDS_DIR` has `build-base`, `build-b0`, `build-b1`: builds of the three commits to compare. + +## Timing rules + +Timing scripts serialize on a lock file (`/tmp/pk-bench.lock`, `flock`), pin cores with +`taskset`, and record the load average at the start of every run. Run them only on a +quiet machine, and keep the load log with the result. Do not quote a timing from a +loaded machine as a benchmark. + +## Slice only head (PR 87) + +`vad_slice_bench.cpp` is built inside the PR 87 tree (it uses internal headers) and driven +by `slice_bench.py speed|load|rss`; see its docstring for the environment variables. +Results are in `results/slice_*`. diff --git a/scripts/vad_bench/analyze_synthetic.py b/scripts/vad_bench/analyze_synthetic.py new file mode 100755 index 0000000..25c6622 --- /dev/null +++ b/scripts/vad_bench/analyze_synthetic.py @@ -0,0 +1,62 @@ +#!/usr/bin/env python3 +"""usage: analyze_synthetic.py synthetic_raw.json -> tables on stdout""" +import json,sys,numpy as np +from heads_lib import mask,counts,prf,kappa,bound_err +R=json.load(open(sys.argv[1])) +conds=R["conds"];refs=R["refs"];P=R["preds"] +dur=lambda i: refs[i][-1][1]+1.0 # unused +def total_sec(i): + return None +import soundfile +# clip length: need exact; recompute from the raw: last pred... use max(ref end, pred end)+2 s margin rounding => use mask length from ref end + 1.5 (trailing silence >=0.5) +def n_sec(i): return max(refs[i][-1][1]+0.5, max([e for s,e in (P['silero_matched'][i] or [(0,0)])]+[0]))+0.0 +order=["clean","white20","white10","white5","white0","pink20","pink10","pink5","pink0"] +def agg(sysname,cs): + c=np.zeros(4);bs=[];be=[];ms=0;me=0;nb=0 + for i,cn in enumerate(conds): + if cn not in cs or P[sysname][i] is None: continue + L=n_sec(i)+3 + ref=mask(refs[i],L);pr=mask(P[sysname][i],L) + c+=counts(pr,ref) + e1,m1=bound_err(P[sysname][i],refs[i],0);e2,m2=bound_err(P[sysname][i],refs[i],1) + bs+=e1;be+=e2;ms+=m1;me+=m2;nb+=len(refs[i]) + return c,bs,be,ms,me,nb +out={} +systems=["silero_matched","silero_default","ultra_q8","redux_packed"] +print("FRAME-LEVEL (10 ms grid), thr 0.5") +print("cond | "+" | ".join(f"{s} P/R/F1" for s in systems)) +rows=[] +for cn in order: + cells=[] + for s in systems: + c,*_=agg(s,[cn]);p,r,f=prf(c);cells.append(f"{p*100:.1f}/{r*100:.1f}/{f*100:.1f}");out[f"{cn}|{s}"]=[p,r,f] + print(cn," | ".join(cells)) +print("\nBOUNDARY ERROR ms (median / p90 / miss%) start;end, pooled") +for cn in ["clean","white10","white0","pink10","pink0"]: + cells=[] + for s in systems: + c,bs,be,ms,me,nb=agg(s,[cn]) + f=lambda a:(f"{np.median(a):.0f}/{np.percentile(a,90):.0f}" if a else "-") + cells.append(f"S {f(bs)} ({100*ms/nb:.1f}%) E {f(be)} ({100*me/nb:.1f}%)") + print(cn," | ".join(cells)) +print("all-conditions boundary:") +for s in systems: + c,bs,be,ms,me,nb=agg(s,order); a=bs+be + print(s,f"median {np.median(a):.0f} p90 {np.percentile(a,90):.0f} start-med {np.median(bs):.0f} end-med {np.median(be):.0f} miss {100*(ms+me)/(2*nb):.1f}%") +print("\nTHRESHOLD SWEEP F1 (clean,white10,white0,pink10 pooled) P/R/F1") +sub=["clean","white10","white0","pink10"] +for base in ["silero_matched","ultra_q8","redux_packed"]: + cells=[] + for t in ("@0.3","","@0.7"): + nm=base+t; c,*_=agg(nm,sub); p,r,f=prf(c); cells.append(f"thr{t or '@0.5'} {p*100:.1f}/{r*100:.1f}/{f*100:.1f}") + print(base," | ".join(cells)) +print("\nAGREEMENT silero_matched vs parakeet (all synthetic): F1 of speech mask, kappa, IoU") +for s in ["ultra_q8","redux_packed"]: + c=np.zeros(4) + for i in range(len(conds)): + L=n_sec(i)+3;c+=counts(mask(P[s][i],L),mask(P["silero_matched"][i],L)) + p,r,f=prf(c);print(s,f"F1 {f*100:.1f} kappa {kappa(c):.3f} IoU {c[0]/(c[0]+c[1]+c[2])*100:.1f}") +c=np.zeros(4) +for i in range(len(conds)): + L=n_sec(i)+3;c+=counts(mask(P["ultra_q8"][i],L),mask(P["redux_packed"][i],L)) +print("ultra vs redux",f"F1 {prf(c)[2]*100:.1f} kappa {kappa(c):.3f}") diff --git a/scripts/vad_bench/compare_synthetic.py b/scripts/vad_bench/compare_synthetic.py new file mode 100755 index 0000000..4b44082 --- /dev/null +++ b/scripts/vad_bench/compare_synthetic.py @@ -0,0 +1,50 @@ +#!/usr/bin/env python3 +"""Synthetic benchmark: Parakeet VAD head (Ultra Q8_0, Redux packed) vs Silero. + +usage: PARAKEET_LIB=build/libparakeet.so compare_synthetic.py DATA_DIR GGUF_DIR OUT.json [CLIPS_PER_COND] +GGUF_DIR holds ultra-q8_0.gguf and redux-keep.gguf. Needs: numpy soundfile torch silero-vad onnxruntime. +Writes the raw spans of every system; analyze_synthetic.py turns them into the tables. +""" +import json,sys,numpy as np,soundfile as sf +from heads_lib import * +SYS={} +def make_systems(): + S={"silero_matched":lambda y:silero(y,True),"silero_default":lambda y:silero(y,False)} + return S +DATA,G,OUT=sys.argv[1],sys.argv[2],sys.argv[3] # data dir, gguf dir, output json +N_CLIPS=int(sys.argv[4]) if len(sys.argv)>4 else 24 # clips per condition +libri=list(np.load(DATA+"/libri.npy",allow_pickle=True)) +rng=np.random.default_rng(7) +order=rng.permutation(len(libri)) +sets=[[libri[i] for i in order[k*5:(k+1)*5]] for k in range(len(libri)//5)][:N_CLIPS] +conds=[("clean",None)]+[(k,s) for k in("white","pink") for s in(20,10,5,0)] +# build corpus once +corpus=[] +for ci,(kind,snr) in enumerate(conds): + for k,us in enumerate(sets): + r=np.random.default_rng(1000+k) # same speech layout per clip across conditions + y,spans=build(us,r) + y=add_noise(y,spans,kind,snr,np.random.default_rng(5000+ci*100+k)) + corpus.append((f"{kind}{'' if snr is None else snr}",y,spans)) +print("corpus",len(corpus),"wavs",sum(len(c[1]) for c in corpus)/16000,"s",flush=True) +models={"ultra_q8":"ultra-q8_0.gguf","redux_packed":"redux-keep.gguf"} +preds={} # system -> list of spans per corpus item +for name,fn in make_systems().items(): + preds[name]=[fn(c[1]) for c in corpus] +print("silero done",flush=True) +for mname,f in models.items(): + pk=PK(G+"/"+f) + preds[mname]=[parakeet(pk,c[1]) for c in corpus] + # threshold sweep on a subset of conditions + for thr in (0.3,0.7): + preds[f"{mname}@{thr}"]=[parakeet(pk,c[1],thr) if c[0] in("clean","white10","white0","pink10") else None for c in corpus] + print(mname,"done",flush=True) +# silero thresholds +for thr in (0.3,0.7): + preds[f"silero_matched@{thr}"]=[None]*len(corpus) + for i,c in enumerate(corpus): + if c[0] in("clean","white10","white0","pink10"): + ts=get_speech_timestamps(torch.from_numpy(c[1]),SIL,sampling_rate=16000,return_seconds=True,threshold=thr,min_speech_duration_ms=100,min_silence_duration_ms=200,speech_pad_ms=0) + preds[f"silero_matched@{thr}"][i]=[(t["start"],t["end"]) for t in ts] +json.dump({"conds":[c[0] for c in corpus],"refs":[c[2] for c in corpus],"preds":preds},open(OUT,"w")) +print("saved",flush=True) diff --git a/scripts/vad_bench/compare_ted.py b/scripts/vad_bench/compare_ted.py new file mode 100755 index 0000000..957f0c5 --- /dev/null +++ b/scripts/vad_bench/compare_ted.py @@ -0,0 +1,51 @@ +#!/usr/bin/env python3 +"""usage: compare_ted.py DATA_DIR GGUF_DIR OUT.json +DATA_DIR holds ted_*.wav and the matching ted_*_words.json from ted_reference.py. +The speed it prints is NOT a valid benchmark (other load); use speed_heads.py.""" +import json,glob,sys,time,subprocess,numpy as np,soundfile as sf +from heads_lib import * +from pk import PK +DATA,G,OUT=sys.argv[1],sys.argv[2],sys.argv[3] +print("uptime before:",subprocess.getoutput("uptime")) +talks=[f for f in sorted(glob.glob(DATA+"/ted_*.wav")) if "S103" not in f] # the original run left out one talk with S103 in its name; reason not recorded +ys={f:sf.read(f,dtype="float32")[0] for f in talks} +refs={} +for f in talks: + W=json.load(open(f.replace(".wav","_words.json"))) + spans=[]; + for s,e in W: + if spans and s-spans[-1][1]<0.4: spans[-1][1]=max(spans[-1][1],e) + else: spans.append([s,e]) + refs[f]=spans +res={};preds={} +def run(name,fn): + cs=np.zeros(4);tt=0;ds=0; + for f in talks: + y=ys[f];t0=time.perf_counter();p=fn(y);dt=time.perf_counter()-t0 + preds[(name,f)]=p;tt+=dt;ds+=len(y)/16000 + L=len(y)/16000+1;cs+=counts(mask(p,L),mask(refs[f],L)) + P,R,F=prf(cs);res[name]=dict(P=P,R=R,F1=F,rtf_x=ds/tt,sec=tt,audio=ds,speech_frac=None) + print(name,"P %.1f R %.1f F1 %.1f VAD speed %.0fx real time (%.1f s for %.0f s)"%(P*100,R*100,F*100,ds/tt,tt,ds),flush=True) +run("silero_matched",lambda y:silero(y,True)) +run("silero_default",lambda y:silero(y,False)) +for n,f in(("ultra_q8","ultra-q8_0.gguf"),("redux_packed","redux-keep.gguf")): + pk=PK(G+"/"+f); run(n,lambda y:parakeet(pk,y)); + out=pk.vad(ys[talks[1]],{"threshold":0.5}) + del pk +# reference speech fraction and agreement +cs=np.zeros(4) +for f in talks: + L=len(ys[f])/16000+1; r=mask(refs[f],L); cs+=np.array([r.sum(),0,0,len(r)-r.sum()]) +print("ref speech fraction %.1f%%"%(100*cs[0]/(cs[0]+cs[3]))) +for n in ["silero_matched","silero_default","ultra_q8","redux_packed"]: + t=sum(mask(preds[(n,f)],len(ys[f])/16000+1).sum() for f in talks);a=sum(len(mask([],len(ys[f])/16000+1)) for f in talks) + print(n,"speech fraction %.1f%%"%(100*t/a)) +print("AGREEMENT on TED talks") +for a,b in(("ultra_q8","silero_matched"),("redux_packed","silero_matched"),("ultra_q8","redux_packed")): + c=np.zeros(4) + for f in talks: + L=len(ys[f])/16000+1;c+=counts(mask(preds[(a,f)],L),mask(preds[(b,f)],L)) + print(a,"vs",b,"F1 %.1f kappa %.3f"%(prf(c)[2]*100,kappa(c))) +print("talks",[(f,round(len(ys[f])/16000)) for f in talks]) +json.dump(res,open(OUT,"w"),indent=1) +print("uptime after:",subprocess.getoutput("uptime")) diff --git a/scripts/vad_bench/fetch_data.py b/scripts/vad_bench/fetch_data.py new file mode 100755 index 0000000..cc4cd0b --- /dev/null +++ b/scripts/vad_bench/fetch_data.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python3 +"""Fetch the public audio used by the VAD benchmarks (streams from the Hugging Face hub). + +usage: fetch_data.py OUT_DIR [--libri-count N] [--ted-seconds S] + +Writes OUT_DIR/libri.npy (LibriSpeech test-clean, every 13th utterance, N of them) and +OUT_DIR/ted_talk.wav (the first TED-LIUM long-form talk longer than 700 s, cut to S seconds, +default 600). The silero parity run used N=200. The vad-compare run used a subset with 24 x 5 +utterances; the script that cut it was not saved, so that corpus is not byte-identical to the +one in the published tables (see docs/vad-benchmarks.md). +""" +import argparse, io, os +import numpy as np, soundfile as sf, librosa +from datasets import load_dataset, Audio + +ap = argparse.ArgumentParser() +ap.add_argument("out") +ap.add_argument("--libri-count", type=int, default=200) +ap.add_argument("--ted-seconds", type=int, default=600) +a = ap.parse_args() +os.makedirs(a.out, exist_ok=True) + +ds = load_dataset("openslr/librispeech_asr", "clean", split="test", streaming=True).cast_column("audio", Audio(decode=False)) +U = [] +for i, ex in enumerate(ds): + if i % 13: + continue + y, sr = sf.read(io.BytesIO(ex["audio"]["bytes"]), dtype="float32") + assert sr == 16000 + U.append(y) + if len(U) >= a.libri_count: + break +np.save(f"{a.out}/libri.npy", np.array(U, dtype=object), allow_pickle=True) +print("libri", len(U)) + +ds = load_dataset("distil-whisper/tedlium-long-form", split="test", streaming=True).cast_column("audio", Audio(decode=False)) +for ex in ds: + y, sr = sf.read(io.BytesIO(ex["audio"]["bytes"]), dtype="float32") + if y.ndim > 1: + y = y.mean(1) + if sr != 16000: + y = librosa.resample(y, orig_sr=sr, target_sr=16000) + if len(y) / 16000 < 700: + continue + sf.write(f"{a.out}/ted_talk.wav", y[: a.ted_seconds * 16000], 16000, subtype="PCM_16") + print("ted", len(y) / 16000) + break diff --git a/scripts/vad_bench/fetch_ted_talks.py b/scripts/vad_bench/fetch_ted_talks.py new file mode 100755 index 0000000..ac2b8dc --- /dev/null +++ b/scripts/vad_bench/fetch_ted_talks.py @@ -0,0 +1,18 @@ +#!/usr/bin/env python3 +"""usage: fetch_ted_talks.py OUT_DIR [N_TALKS] -> OUT_DIR/ted_.wav, the first N talks under 1500 s of +distil-whisper/tedlium-long-form (test). The published run used three talks; which ones is not recorded.""" +import io, re, sys, numpy as np, soundfile as sf, librosa +from datasets import load_dataset, Audio +D=sys.argv[1]; N=int(sys.argv[2]) if len(sys.argv)>2 else 3 +# TED-LIUM long-form: first talks +ds=load_dataset("distil-whisper/tedlium-long-form",split="test",streaming=True).cast_column("audio",Audio(decode=False)) +n=0 +for i,ex in enumerate(ds): + y,sr=sf.read(io.BytesIO(ex["audio"]["bytes"]),dtype="float32") + if y.ndim>1:y=y.mean(1) + if sr!=16000:y=librosa.resample(y,orig_sr=sr,target_sr=16000) + if len(y)/16000>1500: continue + name=re.sub(r"[^A-Za-z0-9_-]","",(ex["audio"]["path"] or f"talk{i}").split("/")[-1].rsplit(".",1)[0]) or f"talk{i}" + sf.write(f"{D}/ted_{name}.wav",y,16000,subtype="PCM_16") + print(name,len(y)/16000,flush=True); n+=1 + if n>=N: break diff --git a/scripts/vad_bench/heads_lib.py b/scripts/vad_bench/heads_lib.py new file mode 100755 index 0000000..dd9d6a3 --- /dev/null +++ b/scripts/vad_bench/heads_lib.py @@ -0,0 +1,60 @@ +"""Detector wrappers for the Parakeet-head vs Silero comparison. +Silero reference: the `silero-vad` pip package (6.2.3), ONNX, one thread. Parakeet: libparakeet via pk.py. +The synthetic-clip helpers are the same code as vad_synth.py (kept apart so each script runs on its own). +""" +import numpy as np, torch +from pk import PK +torch.set_num_threads(1) +from silero_vad import load_silero_vad, get_speech_timestamps +SIL=load_silero_vad(onnx=True) +GR=0.01 # mask grid 10 ms +def mask(spans,n_sec): + m=np.zeros(int(round(n_sec/GR)),bool) + for s,e in spans: + m[int(round(s/GR)):int(round(e/GR))]=True + return m +def silero(y,matched=True): + kw=dict(threshold=0.5,min_speech_duration_ms=100,min_silence_duration_ms=200,speech_pad_ms=0) if matched else {} + ts=get_speech_timestamps(torch.from_numpy(y),SIL,sampling_rate=16000,return_seconds=True,**kw) + return [(t["start"],t["end"]) for t in ts] +def parakeet(pk,y,thr=0.5): + return [(s["start"],s["end"]) for s in pk.vad(y,{"threshold":thr})["segments"]] +def trim(u): + fr=len(u)//160; r=np.sqrt((u[:fr*160].reshape(fr,160)**2).mean(1)) + idx=np.where(r>r.max()*0.01)[0] + return u[idx[0]*160:(idx[-1]+1)*160] +def pink(n,rng): + X=np.fft.rfft(rng.standard_normal(n)); f=np.arange(len(X)); f[0]=1 + y=np.fft.irfft(X/np.sqrt(f),n); return y/y.std() +def build(utts,rng): + g=lambda a,b: rng.uniform(a,b) + parts=[];spans=[];t=g(0.5,1.5) + parts.append(rng.standard_normal(int(t*16000))*1e-3) + for i,u in enumerate(utts): + u=trim(u); spans.append((t,t+len(u)/16000)); parts.append(u); t+=len(u)/16000 + gap=g(0.3,2.0) if i1.0: miss+=1 + else: out.append(d*1000) + return out,miss + diff --git a/scripts/vad_bench/longform_b1.sh b/scripts/vad_bench/longform_b1.sh new file mode 100755 index 0000000..31db519 --- /dev/null +++ b/scripts/vad_bench/longform_b1.sh @@ -0,0 +1,41 @@ +#!/bin/bash +# Batched VAD decode on long talks (PR 82): wall time and peak RSS of `transcribe --vad`. +# usage: longform_b1.sh TALKS_DIR GGUF_DIR BUILDS_DIR OUT_TSV LOADLOG [MAXWAIT_SECONDS] +# TALKS_DIR holds -merged.wav for the four talks named below (TED-LIUM long-form, merged) +# GGUF_DIR holds ultra-q8_0.gguf and redux-keep.gguf +# BUILDS_DIR holds build-base, build-b0, build-b1 (the three builds being compared) +# Runs 5 repetitions per (talk, model, build), build order rotated per repetition, 8 threads pinned +# to cores 0-7, serialized with a lock. Before each run it waits until the mean number of other +# runnable tasks (over 10 s) is below 4. The transcript hash is written to the TSV so equality of +# the outputs can be checked. +T=$1; G=$2; B=$3; R=$4; L=$5; MAXWAIT=${6:-10800}; WAITED=0 +OUTD=$(mktemp -d) +others() { t=0; for i in 1 2 3 4 5 6 7 8 9 10; do n=$(ps -eo stat,comm --no-headers | awk '$1 ~ /^R/ && $2!="ps" && $2!="awk"' | wc -l); t=$((t+n)); sleep 1; done; echo "scale=1; $t/10" | bc; } +wait_quiet() { + while true; do + l=$(cut -d' ' -f1 /proc/loadavg); o=$(others) + if awk "BEGIN{exit !($o < 4)}"; then echo "$(date +%H:%M:%S) ok runnable_mean $o loadavg1 $l" >> "$L"; return 0; fi + echo "$(date +%H:%M:%S) runnable_mean $o loadavg1 $l waiting" >> "$L" + if [ $WAITED -ge $MAXWAIT ]; then echo "GAVE UP after ${WAITED}s" >> "$L"; return 1; fi + sleep 240; WAITED=$((WAITED+250)) + done +} +for talk in BillGates AimeeMullins JaneMcGonigal DanielKahneman; do + for model in ultra-q8_0 redux-keep; do + for run in 1 2 3 4 5; do + builds=(base b0 b1); k=$(( (run-1) % 3 )); builds=("${builds[@]:k}" "${builds[@]:0:k}") + for b in "${builds[@]}"; do + grep -q -P "^$talk\t$model\t$run\t$b\t" "$R" 2>/dev/null && continue + wait_quiet || exit 1 + lb=$(cut -d' ' -f1-3 /proc/loadavg) + flock /tmp/pk-bench.lock taskset -c 0-7 /usr/bin/time -f "%e %M" -o "$OUTD/tm.txt" \ + "$B/build-$b/examples/cli/parakeet-cli" transcribe --model "$G/$model.gguf" --input "$T/$talk-merged.wav" --vad --json --threads 8 > "$OUTD/o.json" 2> "$OUTD/err.txt" + rc=$? + la=$(cut -d' ' -f1-3 /proc/loadavg) + sha=$(sha256sum < "$OUTD/o.json" | cut -c1-16) + echo -e "$talk\t$model\t$run\t$b\t$(tail -1 "$OUTD/tm.txt")\t$sha\trc$rc\tload_before $lb\tload_after $la" >> "$R" + done + done + done +done +rm -rf "$OUTD"; echo ALLDONE >> "$L" diff --git a/scripts/vad_bench/make_clips.py b/scripts/vad_bench/make_clips.py new file mode 100755 index 0000000..8aecc45 --- /dev/null +++ b/scripts/vad_bench/make_clips.py @@ -0,0 +1,31 @@ +#!/usr/bin/env python3 +"""Build the synthetic clips of the Silero parity benchmark. + +usage: make_clips.py DATA_DIR (DATA_DIR holds libri.npy from fetch_data.py) + +Writes DATA_DIR/clips/*.wav and DATA_DIR/clips.json: 40 clips of 5 utterances in 3 conditions +(clean, white noise at 10 dB SNR, pink noise at 10 dB SNR) = 120 clips. The reference speech +spans are the utterance spans (trimmed at 1 percent of peak frame RMS), so they are coarse. +""" +import json, os, sys +import numpy as np, soundfile as sf +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from vad_synth import build, add_noise + +D = sys.argv[1] +os.makedirs(D + "/clips", exist_ok=True) +libri = list(np.load(D + "/libri.npy", allow_pickle=True)) +rng = np.random.default_rng(7) +order = rng.permutation(len(libri)) +sets = [[libri[i] for i in order[k * 5:(k + 1) * 5]] for k in range(len(libri) // 5)] +conds = [("clean", None), ("white", 10), ("pink", 10)] +meta = [] +for ci, (kind, snr) in enumerate(conds): + for k, us in enumerate(sets): + y, spans = build(us, np.random.default_rng(1000 + k)) + y = add_noise(y, spans, kind, snr, np.random.default_rng(5000 + ci * 100 + k)) + name = f"{kind}{'' if snr is None else snr}_{k:02d}" + sf.write(f"{D}/clips/{name}.wav", y, 16000, subtype="PCM_16") + meta.append(dict(name=name, cond=name.split("_")[0], ref=spans, dur=len(y) / 16000)) +json.dump(meta, open(D + "/clips.json", "w")) +print(len(meta), "clips", sum(m["dur"] for m in meta), "s") diff --git a/scripts/vad_bench/onnx_time.py b/scripts/vad_bench/onnx_time.py new file mode 100755 index 0000000..2c1c97d --- /dev/null +++ b/scripts/vad_bench/onnx_time.py @@ -0,0 +1,10 @@ +import sys,time,numpy as np,soundfile as sf,onnxruntime as ort +so=ort.SessionOptions(); so.intra_op_num_threads=1; so.inter_op_num_threads=1 +s=ort.InferenceSession(sys.argv[1],so,providers=["CPUExecutionProvider"]) +y=sf.read(sys.argv[2],dtype="float32")[0]; n=(len(y)+511)//512; y=np.pad(y,(0,n*512-len(y))) +st=np.zeros((2,1,128),np.float32); ctx=np.zeros(64,np.float32); sr=np.array(16000,np.int64) +t0=time.perf_counter() +for i in range(n): + x=np.concatenate([ctx,y[i*512:(i+1)*512]])[None] + o,st=s.run(None,{"input":x,"state":st,"sr":sr}); ctx=x[0,-64:] +print("%.4f"%(time.perf_counter()-t0)) diff --git a/scripts/vad_bench/pk.py b/scripts/vad_bench/pk.py new file mode 100755 index 0000000..9ff0181 --- /dev/null +++ b/scripts/vad_bench/pk.py @@ -0,0 +1,26 @@ +"""ctypes wrapper over libparakeet for the VAD benchmarks. Set PARAKEET_LIB to the path of libparakeet.so.""" +import ctypes, json, os, numpy as np +LIB=os.environ.get("PARAKEET_LIB","build/libparakeet.so") +class PK: + def __init__(self, gguf): + L=self.L=ctypes.CDLL(LIB) + L.parakeet_capi_load.restype=ctypes.c_void_p; L.parakeet_capi_load.argtypes=[ctypes.c_char_p] + for n in ("parakeet_capi_vad_pcm_json","parakeet_capi_vad_path_json","parakeet_capi_transcribe_pcm_batch_json"): + getattr(L,n).restype=ctypes.c_void_p + L.parakeet_capi_vad_pcm_json.argtypes=[ctypes.c_void_p,ctypes.c_void_p,ctypes.c_int,ctypes.c_int,ctypes.c_char_p] + L.parakeet_capi_vad_path_json.argtypes=[ctypes.c_void_p,ctypes.c_char_p,ctypes.c_char_p] + L.parakeet_capi_transcribe_pcm_batch_json.argtypes=[ctypes.c_void_p,ctypes.c_void_p,ctypes.c_void_p,ctypes.c_int,ctypes.c_int,ctypes.c_int] + L.parakeet_capi_free_string.argtypes=[ctypes.c_void_p] + L.parakeet_capi_last_error.restype=ctypes.c_char_p; L.parakeet_capi_last_error.argtypes=[ctypes.c_void_p] + self.ctx=L.parakeet_capi_load(gguf.encode()); assert self.ctx + def _s(self,p): + if not p: raise RuntimeError(self.L.parakeet_capi_last_error(self.ctx).decode()) + s=ctypes.string_at(p).decode(); self.L.parakeet_capi_free_string(p); return s + def vad(self,pcm,opts=None): + pcm=np.ascontiguousarray(pcm,dtype=np.float32) + o=json.dumps(opts).encode() if opts else None + return json.loads(self._s(self.L.parakeet_capi_vad_pcm_json(self.ctx,pcm.ctypes.data,len(pcm),16000,o))) + def words_batch(self,clips): + cat=np.ascontiguousarray(np.concatenate(clips),dtype=np.float32) + n=np.array([len(c) for c in clips],dtype=np.int32) + return json.loads(self._s(self.L.parakeet_capi_transcribe_pcm_batch_json(self.ctx,cat.ctypes.data,n.ctypes.data,len(clips),16000,2))) diff --git a/scripts/vad_bench/results/heads_synthetic_signed.txt b/scripts/vad_bench/results/heads_synthetic_signed.txt new file mode 100644 index 0000000..320cf86 --- /dev/null +++ b/scripts/vad_bench/results/heads_synthetic_signed.txt @@ -0,0 +1,11 @@ +silero_matched start signed median 237 ms, p10 2 p90 548 +silero_matched end signed median 3 ms, p10 -404 p90 124 +silero_default start signed median 232 ms, p10 -46 p90 523 +silero_default end signed median 20 ms, p10 -370 p90 153 +ultra_q8 start signed median 162 ms, p10 -86 p90 469 +ultra_q8 end signed median -10 ms, p10 -420 p90 163 +redux_packed start signed median 147 ms, p10 -91 p90 459 +redux_packed end signed median 54 ms, p10 -332 p90 183 +silero_matched segments per ref span: 1.96 +ultra_q8 segments per ref span: 1.92 +redux_packed segments per ref span: 1.76 diff --git a/scripts/vad_bench/results/heads_synthetic_tables.txt b/scripts/vad_bench/results/heads_synthetic_tables.txt new file mode 100644 index 0000000..17cd523 --- /dev/null +++ b/scripts/vad_bench/results/heads_synthetic_tables.txt @@ -0,0 +1,33 @@ +FRAME-LEVEL (10 ms grid), thr 0.5 +cond | silero_matched P/R/F1 | silero_default P/R/F1 | ultra_q8 P/R/F1 | redux_packed P/R/F1 +clean 99.1/89.7/94.1 | 98.7/90.5/94.4 | 98.2/92.0/95.0 | 98.4/92.7/95.4 +white20 99.4/88.4/93.5 | 99.0/89.2/93.9 | 98.8/89.9/94.1 | 98.7/91.7/95.1 +white10 99.3/88.0/93.3 | 98.9/89.0/93.7 | 98.9/88.3/93.3 | 98.2/91.9/94.9 +white5 99.2/87.8/93.1 | 98.8/88.5/93.4 | 98.9/87.2/92.7 | 97.4/91.9/94.6 +white0 99.2/85.8/92.0 | 98.8/86.7/92.4 | 98.3/87.9/92.8 | 96.4/92.3/94.3 +pink20 99.4/88.4/93.6 | 99.1/89.3/93.9 | 98.8/90.8/94.6 | 98.6/92.0/95.2 +pink10 99.3/88.0/93.3 | 99.0/89.0/93.7 | 99.0/88.8/93.6 | 98.1/92.2/95.1 +pink5 99.4/87.6/93.1 | 99.0/88.5/93.5 | 99.0/88.4/93.4 | 97.4/92.6/94.9 +pink0 99.2/85.2/91.7 | 98.8/85.9/91.9 | 96.3/90.8/93.4 | 95.5/93.2/94.3 + +BOUNDARY ERROR ms (median / p90 / miss%) start;end, pooled +clean S 171/527 (0.0%) E 121/366 (0.0%) | S 162/499 (0.0%) E 137/343 (0.0%) | S 123/460 (0.0%) E 168/325 (0.0%) | S 152/459 (0.0%) E 151/328 (0.0%) +white10 S 249/526 (0.0%) E 116/371 (0.0%) | S 232/500 (0.0%) E 129/357 (0.0%) | S 211/470 (0.0%) E 147/472 (0.8%) | S 163/466 (0.8%) E 152/389 (0.8%) +white0 S 262/571 (0.8%) E 117/459 (2.5%) | S 237/526 (0.8%) E 142/451 (1.7%) | S 162/461 (0.0%) E 149/494 (2.5%) | S 202/460 (0.8%) E 149/404 (1.7%) +pink10 S 248/526 (0.0%) E 106/384 (0.0%) | S 232/502 (0.0%) E 117/343 (0.0%) | S 226/479 (0.0%) E 136/445 (0.0%) | S 176/445 (0.0%) E 148/340 (0.0%) +pink0 S 261/570 (0.0%) E 145/411 (3.3%) | S 248/530 (0.0%) E 150/408 (3.3%) | S 180/457 (2.5%) E 162/470 (2.5%) | S 200/457 (1.7%) E 160/428 (1.7%) +all-conditions boundary: +silero_matched median 143 p90 501 start-med 237 end-med 116 miss 0.4% +silero_default median 149 p90 484 start-med 232 end-med 130 miss 0.3% +ultra_q8 median 154 p90 465 start-med 181 end-med 149 miss 0.6% +redux_packed median 153 p90 437 start-med 170 end-med 149 miss 0.4% + +THRESHOLD SWEEP F1 (clean,white10,white0,pink10 pooled) P/R/F1 +silero_matched thr@0.3 98.5/90.3/94.3 | thr@0.5 99.2/87.9/93.2 | thr@0.7 99.5/86.2/92.4 +ultra_q8 thr@0.3 97.1/92.4/94.7 | thr@0.5 98.6/89.3/93.7 | thr@0.7 99.0/86.2/92.1 +redux_packed thr@0.3 94.8/95.0/94.9 | thr@0.5 97.7/92.3/94.9 | thr@0.7 98.9/90.0/94.2 + +AGREEMENT silero_matched vs parakeet (all synthetic): F1 of speech mask, kappa, IoU +ultra_q8 F1 96.6 kappa 0.882 IoU 93.4 +redux_packed F1 96.3 kappa 0.865 IoU 92.8 +ultra vs redux F1 97.2 kappa 0.896 diff --git a/scripts/vad_bench/results/heads_ted_and_speed_output.txt b/scripts/vad_bench/results/heads_ted_and_speed_output.txt new file mode 100644 index 0000000..4508bcb --- /dev/null +++ b/scripts/vad_bench/results/heads_ted_and_speed_output.txt @@ -0,0 +1,8 @@ +(console output of ted.py, load 6 to 42 during the run, speed numbers there are NOT valid; see speed.py output in the report) +silero_matched P 97.3 R 86.8 F1 91.8 +silero_default P 97.4 R 87.4 F1 92.1 +ultra_q8 P 97.1 R 90.7 F1 93.8 +redux_packed P 97.1 R 91.5 F1 94.2 +ref speech fraction 91.9%; speech fraction silero 82.0 / ultra 85.8 / redux 86.6 +agreement ultra-silero F1 97.0 kappa 0.813; redux-silero F1 96.9 kappa 0.801; ultra-redux F1 98.8 kappa 0.912 +speed.py (300 s of a talk, 4 reps, load 5.8-6.2): silero onnx 1 thr 393x; ultra_q8 216x; ultra_f16 218x; redux_packed 221x; redux_deq_f16 222x (8 threads) diff --git a/scripts/vad_bench/results/longform_b1_loadlog.txt b/scripts/vad_bench/results/longform_b1_loadlog.txt new file mode 100644 index 0000000..21f24b5 --- /dev/null +++ b/scripts/vad_bench/results/longform_b1_loadlog.txt @@ -0,0 +1,127 @@ +11:06:23 load1 12.88 waiting +11:11:23 load1 6.24 waiting +11:16:23 load1 6.73 waiting +11:21:23 load1 16.39 waiting +11:26:23 load1 8.36 waiting +11:32:09 load1 5.57 waiting +11:32:56 ok runnable_mean .1 loadavg1 4.07 (loadavg1 includes our own previous runs) +11:33:50 ok runnable_mean 0 loadavg1 7.06 (loadavg1 includes our own previous runs) +11:34:41 ok runnable_mean 0 loadavg1 8.94 (loadavg1 includes our own previous runs) +11:35:35 ok runnable_mean 0 loadavg1 9.07 (loadavg1 includes our own previous runs) +11:36:25 ok runnable_mean 0 loadavg1 8.73 (loadavg1 includes our own previous runs) +11:37:19 ok runnable_mean 0 loadavg1 8.72 (loadavg1 includes our own previous runs) +11:38:10 ok runnable_mean .1 loadavg1 8.26 (loadavg1 includes our own previous runs) +11:39:05 ok runnable_mean 0 loadavg1 7.93 (loadavg1 includes our own previous runs) +11:39:59 ok runnable_mean 0 loadavg1 8.50 (loadavg1 includes our own previous runs) +11:40:53 ok runnable_mean 0 loadavg1 9.20 (loadavg1 includes our own previous runs) +11:41:47 ok runnable_mean 0 loadavg1 8.62 (loadavg1 includes our own previous runs) +11:42:38 ok runnable_mean .1 loadavg1 7.63 (loadavg1 includes our own previous runs) +11:43:32 ok runnable_mean 0 loadavg1 8.26 (loadavg1 includes our own previous runs) +11:44:25 ok runnable_mean 0 loadavg1 7.83 (loadavg1 includes our own previous runs) +11:45:19 ok runnable_mean .1 loadavg1 8.19 (loadavg1 includes our own previous runs) +11:45:57 ok runnable_mean 0 loadavg1 8.16 (loadavg1 includes our own previous runs) +11:46:36 ok runnable_mean 0 loadavg1 8.47 (loadavg1 includes our own previous runs) +11:47:11 ok runnable_mean 0 loadavg1 7.84 (loadavg1 includes our own previous runs) +11:47:49 ok runnable_mean .1 loadavg1 8.15 (loadavg1 includes our own previous runs) +11:48:25 ok runnable_mean .1 loadavg1 7.90 (loadavg1 includes our own previous runs) +11:49:03 ok runnable_mean .1 loadavg1 8.71 (loadavg1 includes our own previous runs) +11:49:38 ok runnable_mean 0 loadavg1 8.12 (loadavg1 includes our own previous runs) +11:50:15 ok runnable_mean 0 loadavg1 7.81 (loadavg1 includes our own previous runs) +11:50:53 ok runnable_mean .1 loadavg1 8.60 (loadavg1 includes our own previous runs) +11:51:31 ok runnable_mean .1 loadavg1 9.13 (loadavg1 includes our own previous runs) +11:52:09 ok runnable_mean 0 loadavg1 8.25 (loadavg1 includes our own previous runs) +11:52:45 ok runnable_mean .1 loadavg1 7.47 (loadavg1 includes our own previous runs) +11:53:23 ok runnable_mean .1 loadavg1 7.70 (loadavg1 includes our own previous runs) +11:54:00 ok runnable_mean .1 loadavg1 7.57 (loadavg1 includes our own previous runs) +11:54:38 ok runnable_mean 0 loadavg1 7.11 (loadavg1 includes our own previous runs) +11:55:25 ok runnable_mean 0 loadavg1 7.75 (loadavg1 includes our own previous runs) +11:56:12 ok runnable_mean .2 loadavg1 8.74 (loadavg1 includes our own previous runs) +11:56:56 ok runnable_mean .3 loadavg1 8.60 (loadavg1 includes our own previous runs) +11:57:43 ok runnable_mean 0 loadavg1 8.07 (loadavg1 includes our own previous runs) +11:58:28 ok runnable_mean 0 loadavg1 8.62 (loadavg1 includes our own previous runs) +11:59:15 ok runnable_mean .1 loadavg1 8.62 (loadavg1 includes our own previous runs) +11:59:59 ok runnable_mean 0 loadavg1 8.55 (loadavg1 includes our own previous runs) +12:00:48 ok runnable_mean 0 loadavg1 8.71 (loadavg1 includes our own previous runs) +12:01:39 ok runnable_mean 0 loadavg1 8.99 (loadavg1 includes our own previous runs) +12:02:33 ok runnable_mean .2 loadavg1 9.98 (loadavg1 includes our own previous runs) +12:03:20 ok runnable_mean 0 loadavg1 9.46 (loadavg1 includes our own previous runs) +12:04:05 ok runnable_mean .2 loadavg1 8.98 (loadavg1 includes our own previous runs) +12:04:53 ok runnable_mean 0 loadavg1 8.86 (loadavg1 includes our own previous runs) +12:05:39 ok runnable_mean .8 loadavg1 9.07 (loadavg1 includes our own previous runs) +12:06:27 ok runnable_mean 0 loadavg1 8.74 (loadavg1 includes our own previous runs) +12:07:00 ok runnable_mean 0 loadavg1 8.65 (loadavg1 includes our own previous runs) +12:07:34 ok runnable_mean 0 loadavg1 8.55 (loadavg1 includes our own previous runs) +12:08:06 ok runnable_mean 0 loadavg1 8.94 (loadavg1 includes our own previous runs) +12:08:40 ok runnable_mean 0 loadavg1 8.00 (loadavg1 includes our own previous runs) +12:09:13 ok runnable_mean .1 loadavg1 7.60 (loadavg1 includes our own previous runs) +12:09:46 ok runnable_mean .1 loadavg1 7.28 (loadavg1 includes our own previous runs) +12:10:18 ok runnable_mean .1 loadavg1 7.19 (loadavg1 includes our own previous runs) +12:10:52 ok runnable_mean 0 loadavg1 7.59 (loadavg1 includes our own previous runs) +12:11:26 ok runnable_mean 0 loadavg1 8.07 (loadavg1 includes our own previous runs) +12:11:59 ok runnable_mean .1 loadavg1 8.06 (loadavg1 includes our own previous runs) +12:12:32 ok runnable_mean .1 loadavg1 8.03 (loadavg1 includes our own previous runs) +12:13:04 ok runnable_mean .1 loadavg1 8.12 (loadavg1 includes our own previous runs) +12:13:38 ok runnable_mean 0 loadavg1 7.93 (loadavg1 includes our own previous runs) +12:14:11 ok runnable_mean 1.5 loadavg1 8.46 (loadavg1 includes our own previous runs) +12:15:02 runnable_mean 11.3 loadavg1 17.44 waiting +12:19:13 ok runnable_mean .7 loadavg1 3.35 (loadavg1 includes our own previous runs) +12:19:58 ok runnable_mean 1.2 loadavg1 7.84 (loadavg1 includes our own previous runs) +12:20:44 ok runnable_mean .2 loadavg1 8.66 (loadavg1 includes our own previous runs) +12:21:27 ok runnable_mean .2 loadavg1 8.30 (loadavg1 includes our own previous runs) +12:22:13 ok runnable_mean 0 loadavg1 7.92 (loadavg1 includes our own previous runs) +12:22:57 ok runnable_mean .6 loadavg1 7.92 (loadavg1 includes our own previous runs) +12:23:43 ok runnable_mean .1 loadavg1 9.26 (loadavg1 includes our own previous runs) +12:24:27 ok runnable_mean 0 loadavg1 9.19 (loadavg1 includes our own previous runs) +12:25:13 ok runnable_mean 0 loadavg1 8.77 (loadavg1 includes our own previous runs) +12:25:58 ok runnable_mean .3 loadavg1 8.46 (loadavg1 includes our own previous runs) +12:26:44 ok runnable_mean 0 loadavg1 8.66 (loadavg1 includes our own previous runs) +12:27:30 ok runnable_mean .4 loadavg1 8.67 (loadavg1 includes our own previous runs) +12:28:13 ok runnable_mean .1 loadavg1 8.99 (loadavg1 includes our own previous runs) +12:28:58 ok runnable_mean .1 loadavg1 8.40 (loadavg1 includes our own previous runs) +12:29:44 ok runnable_mean .1 loadavg1 8.50 (loadavg1 includes our own previous runs) +12:30:31 ok runnable_mean .1 loadavg1 9.08 (loadavg1 includes our own previous runs) +12:31:06 ok runnable_mean .1 loadavg1 8.62 (loadavg1 includes our own previous runs) +12:31:40 ok runnable_mean .1 loadavg1 8.33 (loadavg1 includes our own previous runs) +12:32:16 ok runnable_mean .3 loadavg1 8.24 (loadavg1 includes our own previous runs) +12:32:51 ok runnable_mean .4 loadavg1 8.10 (loadavg1 includes our own previous runs) +12:33:41 ok runnable_mean 0 loadavg1 9.59 (loadavg1 includes our own previous runs) +12:34:16 ok runnable_mean .1 loadavg1 9.26 (loadavg1 includes our own previous runs) +12:34:49 ok runnable_mean 0 loadavg1 9.09 (loadavg1 includes our own previous runs) +12:35:24 ok runnable_mean .5 loadavg1 11.79 (loadavg1 includes our own previous runs) +12:35:57 ok runnable_mean .3 loadavg1 10.43 (loadavg1 includes our own previous runs) +12:36:30 ok runnable_mean .4 loadavg1 9.94 (loadavg1 includes our own previous runs) +12:37:03 ok runnable_mean .1 loadavg1 9.55 (loadavg1 includes our own previous runs) +12:37:33 ok runnable_mean 0 loadavg1 8.41 (loadavg1 includes our own previous runs) +12:38:06 ok runnable_mean .4 loadavg1 8.41 (loadavg1 includes our own previous runs) +12:38:37 ok runnable_mean .8 loadavg1 9.80 (loadavg1 includes our own previous runs) +12:39:11 ok runnable_mean .7 loadavg1 11.22 (loadavg1 includes our own previous runs) +12:39:53 ok runnable_mean .3 loadavg1 10.11 (loadavg1 includes our own previous runs) +12:40:39 ok runnable_mean 0 loadavg1 11.12 (loadavg1 includes our own previous runs) +12:41:23 ok runnable_mean .2 loadavg1 11.19 (loadavg1 includes our own previous runs) +12:42:06 ok runnable_mean 0 loadavg1 10.11 (loadavg1 includes our own previous runs) +12:42:51 ok runnable_mean 0 loadavg1 10.49 (loadavg1 includes our own previous runs) +12:43:37 ok runnable_mean .2 loadavg1 11.10 (loadavg1 includes our own previous runs) +12:44:20 ok runnable_mean .8 loadavg1 11.09 (loadavg1 includes our own previous runs) +12:45:04 ok runnable_mean .7 loadavg1 11.22 (loadavg1 includes our own previous runs) +12:45:47 ok runnable_mean .1 loadavg1 10.74 (loadavg1 includes our own previous runs) +12:46:30 ok runnable_mean .1 loadavg1 10.03 (loadavg1 includes our own previous runs) +12:47:13 ok runnable_mean .1 loadavg1 9.77 (loadavg1 includes our own previous runs) +12:47:55 ok runnable_mean 0 loadavg1 9.08 (loadavg1 includes our own previous runs) +12:48:38 ok runnable_mean .1 loadavg1 9.35 (loadavg1 includes our own previous runs) +12:49:19 ok runnable_mean 0 loadavg1 9.11 (loadavg1 includes our own previous runs) +12:50:02 ok runnable_mean 0 loadavg1 8.97 (loadavg1 includes our own previous runs) +12:50:32 ok runnable_mean .3 loadavg1 8.62 (loadavg1 includes our own previous runs) +12:51:04 ok runnable_mean .4 loadavg1 9.29 (loadavg1 includes our own previous runs) +12:51:35 ok runnable_mean 1.1 loadavg1 9.50 (loadavg1 includes our own previous runs) +12:52:07 ok runnable_mean .4 loadavg1 10.83 (loadavg1 includes our own previous runs) +12:52:46 ok runnable_mean 1.0 loadavg1 18.00 (loadavg1 includes our own previous runs) +12:53:20 ok runnable_mean 1.1 loadavg1 17.59 (loadavg1 includes our own previous runs) +12:53:51 ok runnable_mean .1 loadavg1 14.59 (loadavg1 includes our own previous runs) +12:54:22 ok runnable_mean 1.1 loadavg1 11.88 (loadavg1 includes our own previous runs) +12:54:57 ok runnable_mean .1 loadavg1 12.34 (loadavg1 includes our own previous runs) +12:55:27 ok runnable_mean 0 loadavg1 10.23 (loadavg1 includes our own previous runs) +12:55:57 ok runnable_mean .3 loadavg1 9.00 (loadavg1 includes our own previous runs) +12:56:26 ok runnable_mean .1 loadavg1 9.82 (loadavg1 includes our own previous runs) +12:56:58 ok runnable_mean .2 loadavg1 9.08 (loadavg1 includes our own previous runs) +12:57:33 ok runnable_mean 1.1 loadavg1 9.67 (loadavg1 includes our own previous runs) +ALLDONE diff --git a/scripts/vad_bench/results/longform_b1_results.tsv b/scripts/vad_bench/results/longform_b1_results.tsv new file mode 100644 index 0000000..6ba132b --- /dev/null +++ b/scripts/vad_bench/results/longform_b1_results.tsv @@ -0,0 +1,120 @@ +BillGates ultra-q8_0 1 base 45.96 2053092 69a20c09ad148e2f rc0 load_before 2.49 7.32 11.31 load_after 5.57 7.42 11.15 +BillGates ultra-q8_0 1 b0 43.41 2053224 69a20c09ad148e2f rc0 load_before 3.67 6.67 10.72 load_after 7.06 7.16 10.69 +BillGates ultra-q8_0 1 b1 41.24 2087760 69a20c09ad148e2f rc0 load_before 6.29 6.99 10.60 load_after 8.94 7.57 10.64 +BillGates ultra-q8_0 2 b0 43.74 2053060 69a20c09ad148e2f rc0 load_before 8.03 7.42 10.56 load_after 9.07 7.77 10.53 +BillGates ultra-q8_0 2 b1 39.54 2087804 69a20c09ad148e2f rc0 load_before 8.06 7.60 10.44 load_after 8.73 7.80 10.38 +BillGates ultra-q8_0 2 base 44.17 2053448 69a20c09ad148e2f rc0 load_before 7.70 7.61 10.29 load_after 8.72 7.90 10.26 +BillGates ultra-q8_0 3 b1 40.81 2088132 69a20c09ad148e2f rc0 load_before 7.69 7.70 10.17 load_after 8.26 7.80 10.10 +BillGates ultra-q8_0 3 base 44.72 2053220 69a20c09ad148e2f rc0 load_before 7.22 7.59 10.00 load_after 7.93 7.70 9.92 +BillGates ultra-q8_0 3 b0 44.35 2053268 69a20c09ad148e2f rc0 load_before 6.86 7.48 9.83 load_after 8.50 7.82 9.83 +BillGates ultra-q8_0 4 base 43.47 2053460 69a20c09ad148e2f rc0 load_before 7.81 7.70 9.77 load_after 9.20 8.10 9.81 +BillGates ultra-q8_0 4 b0 43.73 2053512 69a20c09ad148e2f rc0 load_before 7.94 7.86 9.71 load_after 8.62 8.03 9.69 +BillGates ultra-q8_0 4 b1 40.77 2087232 69a20c09ad148e2f rc0 load_before 7.59 7.83 9.60 load_after 7.63 7.77 9.50 +BillGates ultra-q8_0 5 b0 43.56 2053400 69a20c09ad148e2f rc0 load_before 6.69 7.56 9.41 load_after 8.26 7.84 9.43 +BillGates ultra-q8_0 5 b1 42.99 2088048 69a20c09ad148e2f rc0 load_before 7.15 7.62 9.34 load_after 7.83 7.71 9.28 +BillGates ultra-q8_0 5 base 43.58 2053220 69a20c09ad148e2f rc0 load_before 6.78 7.49 9.19 load_after 8.19 7.74 9.20 +BillGates redux-keep 1 base 28.11 1514576 7b7ee8115e0d94db rc0 load_before 7.24 7.55 9.12 load_after 8.16 7.75 9.14 +BillGates redux-keep 1 b0 28.55 1514820 7b7ee8115e0d94db rc0 load_before 7.81 7.69 9.11 load_after 8.47 7.84 9.11 +BillGates redux-keep 1 b1 25.00 1547740 7b7ee8115e0d94db rc0 load_before 7.48 7.64 9.03 load_after 7.84 7.71 9.02 +BillGates redux-keep 2 b0 27.74 1514676 7b7ee8115e0d94db rc0 load_before 6.94 7.52 8.94 load_after 8.15 7.76 8.98 +BillGates redux-keep 2 b1 26.02 1548692 7b7ee8115e0d94db rc0 load_before 7.20 7.57 8.90 load_after 7.90 7.69 8.91 +BillGates redux-keep 2 base 27.32 1514580 7b7ee8115e0d94db rc0 load_before 7.07 7.52 8.84 load_after 8.71 7.88 8.92 +BillGates redux-keep 3 b1 25.00 1547820 7b7ee8115e0d94db rc0 load_before 7.76 7.70 8.85 load_after 8.12 7.78 8.84 +BillGates redux-keep 3 base 27.43 1514780 7b7ee8115e0d94db rc0 load_before 7.10 7.57 8.76 load_after 7.81 7.69 8.77 +BillGates redux-keep 3 b0 27.13 1515028 7b7ee8115e0d94db rc0 load_before 7.73 7.68 8.76 load_after 8.60 7.90 8.80 +BillGates redux-keep 4 base 28.26 1514612 7b7ee8115e0d94db rc0 load_before 7.58 7.71 8.72 load_after 9.13 8.06 8.81 +BillGates redux-keep 4 b0 27.50 1514728 7b7ee8115e0d94db rc0 load_before 8.04 7.86 8.74 load_after 8.25 7.92 8.73 +BillGates redux-keep 4 b1 25.90 1546044 7b7ee8115e0d94db rc0 load_before 7.30 7.72 8.66 load_after 7.47 7.73 8.64 +BillGates redux-keep 5 b0 28.09 1514744 7b7ee8115e0d94db rc0 load_before 6.71 7.56 8.57 load_after 7.70 7.70 8.58 +BillGates redux-keep 5 b1 26.51 1548476 7b7ee8115e0d94db rc0 load_before 6.74 7.50 8.51 load_after 7.57 7.62 8.52 +BillGates redux-keep 5 base 28.25 1514496 7b7ee8115e0d94db rc0 load_before 6.71 7.44 8.45 load_after 7.11 7.46 8.43 +AimeeMullins ultra-q8_0 1 base 36.12 2005520 bc4bf944d41e6aca rc0 load_before 6.47 7.31 8.37 load_after 7.75 7.52 8.40 +AimeeMullins ultra-q8_0 1 b0 36.72 2005432 bc4bf944d41e6aca rc0 load_before 7.66 7.51 8.38 load_after 8.74 7.77 8.44 +AimeeMullins ultra-q8_0 1 b1 34.58 2040184 bc4bf944d41e6aca rc0 load_before 7.79 7.60 8.38 load_after 8.60 7.79 8.41 +AimeeMullins ultra-q8_0 2 b0 36.17 2005260 bc4bf944d41e6aca rc0 load_before 7.51 7.59 8.34 load_after 8.07 7.76 8.37 +AimeeMullins ultra-q8_0 2 b1 34.83 2040348 bc4bf944d41e6aca rc0 load_before 7.06 7.55 8.29 load_after 8.62 7.84 8.36 +AimeeMullins ultra-q8_0 2 base 36.74 2005544 bc4bf944d41e6aca rc0 load_before 7.76 7.68 8.30 load_after 8.62 7.88 8.34 +AimeeMullins ultra-q8_0 3 b1 34.47 2040380 bc4bf944d41e6aca rc0 load_before 7.69 7.70 8.28 load_after 8.55 7.92 8.33 +AimeeMullins ultra-q8_0 3 base 38.42 2005180 bc4bf944d41e6aca rc0 load_before 7.46 7.71 8.26 load_after 8.71 8.00 8.34 +AimeeMullins ultra-q8_0 3 b0 40.66 2005176 bc4bf944d41e6aca rc0 load_before 7.83 7.84 8.28 load_after 8.99 8.11 8.35 +AimeeMullins ultra-q8_0 4 base 44.01 2005284 bc4bf944d41e6aca rc0 load_before 7.91 7.91 8.28 load_after 9.98 8.43 8.44 +AimeeMullins ultra-q8_0 4 b0 36.86 2005304 bc4bf944d41e6aca rc0 load_before 8.83 8.24 8.38 load_after 9.46 8.44 8.44 +AimeeMullins ultra-q8_0 4 b1 35.10 2040732 bc4bf944d41e6aca rc0 load_before 8.39 8.24 8.38 load_after 8.98 8.41 8.43 +AimeeMullins ultra-q8_0 5 b0 37.64 2005480 bc4bf944d41e6aca rc0 load_before 7.83 8.18 8.35 load_after 8.86 8.41 8.42 +AimeeMullins ultra-q8_0 5 b1 36.13 2040572 bc4bf944d41e6aca rc0 load_before 8.11 8.26 8.37 load_after 9.07 8.48 8.44 +AimeeMullins ultra-q8_0 5 base 36.91 2005372 bc4bf944d41e6aca rc0 load_before 8.05 8.28 8.38 load_after 8.74 8.42 8.42 +AimeeMullins redux-keep 1 base 23.40 1466388 0713636b1d7ffa09 rc0 load_before 7.70 8.21 8.35 load_after 8.65 8.39 8.41 +AimeeMullins redux-keep 1 b0 23.82 1466736 0713636b1d7ffa09 rc0 load_before 7.47 8.14 8.33 load_after 8.55 8.34 8.39 +AimeeMullins redux-keep 1 b1 21.84 1502588 0713636b1d7ffa09 rc0 load_before 8.02 8.23 8.35 load_after 8.94 8.43 8.42 +AimeeMullins redux-keep 2 b0 23.67 1466420 0713636b1d7ffa09 rc0 load_before 8.18 8.28 8.37 load_after 8.00 8.23 8.35 +AimeeMullins redux-keep 2 b1 22.31 1502592 0713636b1d7ffa09 rc0 load_before 7.16 8.04 8.29 load_after 7.60 8.07 8.29 +AimeeMullins redux-keep 2 base 23.41 1466976 0713636b1d7ffa09 rc0 load_before 6.58 7.84 8.21 load_after 7.28 7.89 8.22 +AimeeMullins redux-keep 3 b1 21.89 1503932 0713636b1d7ffa09 rc0 load_before 6.54 7.71 8.16 load_after 7.19 7.76 8.16 +AimeeMullins redux-keep 3 base 23.51 1466476 0713636b1d7ffa09 rc0 load_before 6.63 7.62 8.11 load_after 7.59 7.77 8.15 +AimeeMullins redux-keep 3 b0 23.63 1466228 0713636b1d7ffa09 rc0 load_before 6.73 7.58 8.08 load_after 8.07 7.81 8.15 +AimeeMullins redux-keep 4 base 22.96 1466516 0713636b1d7ffa09 rc0 load_before 6.98 7.59 8.07 load_after 8.06 7.80 8.13 +AimeeMullins redux-keep 4 b0 22.94 1466748 0713636b1d7ffa09 rc0 load_before 7.13 7.61 8.06 load_after 8.03 7.78 8.11 +AimeeMullins redux-keep 4 b1 21.93 1502564 0713636b1d7ffa09 rc0 load_before 7.03 7.57 8.04 load_after 8.12 7.79 8.10 +AimeeMullins redux-keep 5 b0 23.79 1466556 0713636b1d7ffa09 rc0 load_before 7.26 7.61 8.04 load_after 7.93 7.73 8.06 +AimeeMullins redux-keep 5 b1 22.61 1503800 0713636b1d7ffa09 rc0 load_before 7.02 7.54 8.00 load_after 8.46 7.80 8.07 +AimeeMullins redux-keep 5 base 41.30 1466792 0713636b1d7ffa09 rc0 load_before 7.61 7.64 8.02 load_after 17.44 9.90 8.75 +JaneMcGonigal ultra-q8_0 1 base 35.54 1990188 c373acef41ee4baa rc0 load_before 4.92 7.27 8.03 load_after 7.84 7.74 8.16 +JaneMcGonigal ultra-q8_0 1 b0 35.32 1989888 c373acef41ee4baa rc0 load_before 7.01 7.56 8.10 load_after 8.66 7.91 8.20 +JaneMcGonigal ultra-q8_0 1 b1 33.31 2016792 c373acef41ee4baa rc0 load_before 7.93 7.78 8.15 load_after 8.30 7.87 8.17 +JaneMcGonigal ultra-q8_0 2 b0 34.92 1990392 c373acef41ee4baa rc0 load_before 7.41 7.70 8.11 load_after 7.92 7.77 8.12 +JaneMcGonigal ultra-q8_0 2 b1 33.95 2016712 c373acef41ee4baa rc0 load_before 6.86 7.54 8.04 load_after 7.92 7.69 8.07 +JaneMcGonigal ultra-q8_0 2 base 35.89 1989796 c373acef41ee4baa rc0 load_before 7.17 7.53 8.01 load_after 9.26 8.01 8.15 +JaneMcGonigal ultra-q8_0 3 b1 34.47 2016736 c373acef41ee4baa rc0 load_before 7.81 7.75 8.07 load_after 9.19 8.08 8.17 +JaneMcGonigal ultra-q8_0 3 base 34.97 1990160 c373acef41ee4baa rc0 load_before 8.00 7.86 8.09 load_after 8.77 8.06 8.15 +JaneMcGonigal ultra-q8_0 3 b0 34.86 1990232 c373acef41ee4baa rc0 load_before 7.65 7.84 8.08 load_after 8.46 8.01 8.12 +JaneMcGonigal ultra-q8_0 4 base 36.67 1990216 c373acef41ee4baa rc0 load_before 7.38 7.79 8.05 load_after 8.66 8.03 8.12 +JaneMcGonigal ultra-q8_0 4 b0 34.96 1989868 c373acef41ee4baa rc0 load_before 7.64 7.83 8.06 load_after 8.67 8.07 8.13 +JaneMcGonigal ultra-q8_0 4 b1 32.92 2017000 c373acef41ee4baa rc0 load_before 7.65 7.87 8.06 load_after 8.99 8.17 8.16 +JaneMcGonigal ultra-q8_0 5 b0 34.68 1989776 c373acef41ee4baa rc0 load_before 7.92 7.97 8.09 load_after 8.40 8.06 8.12 +JaneMcGonigal ultra-q8_0 5 b1 35.70 2016752 c373acef41ee4baa rc0 load_before 7.33 7.84 8.05 load_after 8.50 8.03 8.10 +JaneMcGonigal ultra-q8_0 5 base 37.53 1990160 c373acef41ee4baa rc0 load_before 7.50 7.83 8.03 load_after 9.08 8.19 8.14 +JaneMcGonigal redux-keep 1 base 24.12 1451392 3978446d6aac4afe rc0 load_before 8.06 8.00 8.08 load_after 8.62 8.13 8.12 +JaneMcGonigal redux-keep 1 b0 24.67 1451404 3978446d6aac4afe rc0 load_before 7.90 7.99 8.08 load_after 8.33 8.08 8.11 +JaneMcGonigal redux-keep 1 b1 25.57 1482340 3978446d6aac4afe rc0 load_before 7.43 7.89 8.04 load_after 8.24 8.03 8.09 +JaneMcGonigal redux-keep 2 b0 24.52 1451060 3978446d6aac4afe rc0 load_before 7.43 7.87 8.03 load_after 8.10 7.97 8.06 +JaneMcGonigal redux-keep 2 b1 39.68 1483788 3978446d6aac4afe rc0 load_before 7.48 7.83 8.02 load_after 9.59 8.33 8.17 +JaneMcGonigal redux-keep 2 base 24.96 1451660 3978446d6aac4afe rc0 load_before 8.73 8.19 8.13 load_after 9.26 8.34 8.18 +JaneMcGonigal redux-keep 3 b1 23.02 1483472 3978446d6aac4afe rc0 load_before 8.15 8.13 8.11 load_after 9.09 8.36 8.19 +JaneMcGonigal redux-keep 3 base 24.44 1450932 3978446d6aac4afe rc0 load_before 8.15 8.18 8.14 load_after 11.79 9.03 8.42 +JaneMcGonigal redux-keep 3 b0 23.15 1451448 3978446d6aac4afe rc0 load_before 10.29 8.80 8.35 load_after 10.43 8.92 8.40 +JaneMcGonigal redux-keep 4 base 22.72 1451568 3978446d6aac4afe rc0 load_before 9.05 8.67 8.32 load_after 9.94 8.92 8.42 +JaneMcGonigal redux-keep 4 b0 22.90 1451504 3978446d6aac4afe rc0 load_before 8.88 8.73 8.36 load_after 9.55 8.88 8.42 +JaneMcGonigal redux-keep 4 b1 20.00 1483192 3978446d6aac4afe rc0 load_before 8.31 8.64 8.34 load_after 8.41 8.63 8.35 +JaneMcGonigal redux-keep 5 b0 22.10 1451504 3978446d6aac4afe rc0 load_before 7.43 8.41 8.28 load_after 8.41 8.57 8.33 +JaneMcGonigal redux-keep 5 b1 21.51 1485412 3978446d6aac4afe rc0 load_before 7.51 8.37 8.27 load_after 9.80 8.81 8.42 +JaneMcGonigal redux-keep 5 base 22.87 1451368 3978446d6aac4afe rc0 load_before 9.15 8.70 8.38 load_after 11.22 9.26 8.58 +DanielKahneman ultra-q8_0 1 base 32.65 1974592 669e94614b0cba3a rc0 load_before 9.87 9.03 8.51 load_after 10.11 9.18 8.58 +DanielKahneman ultra-q8_0 1 b0 35.73 1974896 669e94614b0cba3a rc0 load_before 8.87 8.94 8.51 load_after 11.12 9.45 8.69 +DanielKahneman ultra-q8_0 1 b1 33.08 2007260 669e94614b0cba3a rc0 load_before 9.86 9.24 8.63 load_after 11.19 9.63 8.78 +DanielKahneman ultra-q8_0 2 b0 33.67 1974456 669e94614b0cba3a rc0 load_before 9.93 9.41 8.72 load_after 10.11 9.51 8.78 +DanielKahneman ultra-q8_0 2 b1 34.68 2007480 669e94614b0cba3a rc0 load_before 9.91 9.49 8.78 load_after 10.49 9.66 8.87 +DanielKahneman ultra-q8_0 2 base 35.30 1974240 669e94614b0cba3a rc0 load_before 9.35 9.44 8.80 load_after 11.10 9.87 8.97 +DanielKahneman ultra-q8_0 3 b1 32.86 2007008 669e94614b0cba3a rc0 load_before 10.18 9.71 8.93 load_after 11.09 9.98 9.05 +DanielKahneman ultra-q8_0 3 base 34.12 1974776 669e94614b0cba3a rc0 load_before 9.92 9.77 8.99 load_after 11.22 10.10 9.13 +DanielKahneman ultra-q8_0 3 b0 33.07 1974372 669e94614b0cba3a rc0 load_before 10.17 9.91 9.08 load_after 10.74 10.08 9.16 +DanielKahneman ultra-q8_0 4 base 32.30 1974504 669e94614b0cba3a rc0 load_before 9.33 9.80 9.08 load_after 10.03 9.93 9.15 +DanielKahneman ultra-q8_0 4 b0 33.05 1974652 669e94614b0cba3a rc0 load_before 9.16 9.75 9.10 load_after 9.77 9.87 9.16 +DanielKahneman ultra-q8_0 4 b1 31.43 2007432 669e94614b0cba3a rc0 load_before 8.50 9.59 9.08 load_after 9.08 9.65 9.12 +DanielKahneman ultra-q8_0 5 b0 32.67 1974588 669e94614b0cba3a rc0 load_before 8.07 9.42 9.05 load_after 9.35 9.58 9.11 +DanielKahneman ultra-q8_0 5 b1 31.29 2007608 669e94614b0cba3a rc0 load_before 8.14 9.31 9.03 load_after 9.11 9.39 9.07 +DanielKahneman ultra-q8_0 5 base 32.18 1974064 669e94614b0cba3a rc0 load_before 7.87 9.12 8.98 load_after 8.97 9.28 9.04 +DanielKahneman redux-keep 1 base 20.55 1437716 0c96f14edad70cd7 rc0 load_before 8.31 9.12 8.99 load_after 8.62 9.13 9.00 +DanielKahneman redux-keep 1 b0 21.47 1437936 0c96f14edad70cd7 rc0 load_before 7.82 8.95 8.94 load_after 9.29 9.19 9.02 +DanielKahneman redux-keep 1 b1 21.19 1470068 0c96f14edad70cd7 rc0 load_before 8.70 9.07 8.98 load_after 9.50 9.23 9.04 +DanielKahneman redux-keep 2 b0 21.65 1435688 0c96f14edad70cd7 rc0 load_before 8.66 9.06 8.98 load_after 10.83 9.51 9.13 +DanielKahneman redux-keep 2 b1 29.04 1469756 0c96f14edad70cd7 rc0 load_before 12.04 9.80 9.23 load_after 18.00 11.42 9.79 +DanielKahneman redux-keep 2 base 23.07 1434520 0c96f14edad70cd7 rc0 load_before 18.00 11.64 9.87 load_after 17.59 12.10 10.08 +DanielKahneman redux-keep 3 b1 20.79 1469588 0c96f14edad70cd7 rc0 load_before 16.32 12.01 10.07 load_after 14.59 11.89 10.07 +DanielKahneman redux-keep 3 base 21.41 1435020 0c96f14edad70cd7 rc0 load_before 12.64 11.56 9.99 load_after 11.88 11.46 9.99 +DanielKahneman redux-keep 3 b0 24.15 1438240 0c96f14edad70cd7 rc0 load_before 10.66 11.21 9.92 load_after 12.34 11.58 10.08 +DanielKahneman redux-keep 4 base 20.22 1434756 0c96f14edad70cd7 rc0 load_before 10.91 11.30 10.00 load_after 10.23 11.10 9.97 +DanielKahneman redux-keep 4 b0 20.04 1437892 0c96f14edad70cd7 rc0 load_before 9.12 10.83 9.89 load_after 9.00 10.68 9.86 +DanielKahneman redux-keep 4 b1 19.09 1469668 0c96f14edad70cd7 rc0 load_before 7.92 10.40 9.78 load_after 9.82 10.65 9.87 +DanielKahneman redux-keep 5 b0 21.08 1437192 0c96f14edad70cd7 rc0 load_before 8.78 10.40 9.80 load_after 9.08 10.35 9.79 +DanielKahneman redux-keep 5 b1 24.75 1469552 0c96f14edad70cd7 rc0 load_before 7.92 10.05 9.70 load_after 9.67 10.31 9.80 +DanielKahneman redux-keep 5 base 23.87 1434984 0c96f14edad70cd7 rc0 load_before 8.72 10.08 9.73 load_after 11.06 10.50 9.88 diff --git a/scripts/vad_bench/results/silero_parity_results.txt b/scripts/vad_bench/results/silero_parity_results.txt new file mode 100644 index 0000000..8d4ab81 --- /dev/null +++ b/scripts/vad_bench/results/silero_parity_results.txt @@ -0,0 +1,50 @@ +PROBABILITY PARITY vs onnxruntime (v6.2.3 ONNX, official 64-sample context), all 120 clips + 600 s talk, per 32 ms chunk +system | max abs diff | mean abs diff | frames > 0.05 off | decision agreement @0.5 | flipped frames +wcpp_conv623 | 0.7373 | 0.00224 | 0.54% | 99.826% | 309 +wcpp_repo620 | 0.7373 | 0.00224 | 0.54% | 99.826% | 309 +pk_f32 | 0.0001 | 0.00002 | 0.00% | 99.999% | 2 +pk_f16 | 0.0036 | 0.00005 | 0.00% | 99.997% | 5 +pairwise (whisper.cpp converted v6.2.3 vs parakeet f32): +wcpp_conv623 vs pk_f32 | 0.7373 | 0.00224 | 0.54% | 99.825% | 311 +wcpp_conv623 vs wcpp_repo620 | 0.0000 | 0.00000 | 0.00% | 100.000% | 0 +per-region parity vs onnx (mean abs diff) : first chunk of each clip / rest +wcpp_conv623 first-chunk 0.00088 rest 0.00212 +wcpp_repo620 first-chunk 0.00088 rest 0.00212 +pk_f32 first-chunk 0.00002 rest 0.00002 +pk_f16 first-chunk 0.00005 rest 0.00005 + +SEGMENT QUALITY, frame level on 10 ms grid vs synthetic reference (P/R/F1 %), 40 clips per condition +matched post = thr 0.5 / neg 0.35 / min speech 100 ms / min silence 200 ms / pad 0 (same python code on every probability stream) +system | clean | white10 | pink10 | all +onnx [matched post] | 99.3/89.3/94.0 | 99.4/87.8/93.2 | 99.5/87.4/93.1 | 99.4/88.2/93.5 +wcpp_conv623 [matched post] | 99.3/89.4/94.1 | 99.4/87.8/93.3 | 99.5/87.4/93.0 | 99.4/88.2/93.5 +pk_f32 [matched post] | 99.3/89.3/94.0 | 99.4/87.8/93.2 | 99.5/87.4/93.1 | 99.4/88.2/93.5 +pk_f16 [matched post] | 99.3/89.3/94.0 | 99.4/87.8/93.2 | 99.5/87.4/93.1 | 99.4/88.2/93.5 +wcpp_conv623 [whisper.cpp own default] | 98.9/90.3/94.4 | 99.2/89.0/93.8 | 99.3/88.6/93.6 | 99.1/89.3/94.0 +wcpp_conv623 [whisper.cpp C++ post, matched params] | 99.3/89.4/94.1 | 99.4/87.8/93.3 | 99.5/87.4/93.0 | 99.4/88.2/93.5 +pk_f32 [parakeet own default] | 99.1/89.3/93.9 | 99.3/87.6/93.1 | 99.4/87.3/93.0 | 99.3/88.1/93.3 + +BOUNDARY ERROR vs reference, all conditions pooled: start median/p90, end median/p90, miss% +onnx [matched post] | start 241/553 | end 108/396 | miss 0.2% +wcpp_conv623 [matched post] | start 241/553 | end 108/396 | miss 0.2% +pk_f32 [matched post] | start 241/553 | end 108/396 | miss 0.2% +pk_f16 [matched post] | start 241/553 | end 108/396 | miss 0.2% +wcpp_conv623 [whisper.cpp own default] | start 209/521 | end 124/367 | miss 0.0% +wcpp_conv623 [whisper.cpp C++ post, matched params] | start 237/551 | end 110/398 | miss 0.2% +pk_f32 [parakeet own default] | start 211/523 | end 116/392 | miss 0.0% + +AGREEMENT of speech masks (matched post), all 120 clips pooled: F1 / kappa / IoU +wcpp_conv623 vs pk_f32 | F1 99.92 | kappa 0.9969 | IoU 99.84 +wcpp_conv623 vs onnx | F1 99.92 | kappa 0.9969 | IoU 99.84 +pk_f32 vs onnx | F1 100.00 | kappa 1.0000 | IoU 100.00 +pk_f16 vs pk_f32 | F1 100.00 | kappa 0.9999 | IoU 100.00 +wcpp_conv623 vs wcpp_repo620 | F1 100.00 | kappa 1.0000 | IoU 100.00 + +AGREEMENT on the 600 s TED talk (no reference): F1 / kappa, speech fraction +onnx speech fraction 79.1% +wcpp_conv623 speech fraction 78.8% +pk_f32 speech fraction 79.1% +wcpp_conv623 vs pk_f32 | F1 99.75 | kappa 0.9882 +wcpp_conv623 vs onnx | F1 99.75 | kappa 0.9882 +pk_f32 vs onnx | F1 100.00 | kappa 1.0000 +TED talk flipped frames @0.5 vs onnx: wcpp_conv623 57/18750, wcpp_repo620 57/18750, pk_f32 0/18750, pk_f16 1/18750 \ No newline at end of file diff --git a/scripts/vad_bench/results/silero_speed2_raw.json b/scripts/vad_bench/results/silero_speed2_raw.json new file mode 100644 index 0000000..a3cc405 --- /dev/null +++ b/scripts/vad_bench/results/silero_speed2_raw.json @@ -0,0 +1,110 @@ +[ + { + "job": "onnxruntime_python_1thr_detect_only", + "wall": 0.8058516788296402, + "inproc": 0.6678, + "rss_kb": 106744, + "user": 0.76, + "mean_runnable": 4.8, + "load1": 4.9, + "quiet": true, + "attempts": 1, + "round": 0 + }, + { + "job": "whisper-cli_tiny.en_novad", + "wall": 15.08737738407217, + "inproc": null, + "rss_kb": 296592, + "user": 14.98, + "mean_runnable": 4.0, + "load1": 4.69, + "quiet": true, + "attempts": 1, + "round": 0 + }, + { + "job": "whisper-cli_tiny.en_vad", + "wall": 19.682566916104406, + "inproc": null, + "rss_kb": 314876, + "user": 18.18, + "mean_runnable": 4.4, + "load1": 4.59, + "quiet": true, + "attempts": 1, + "round": 0 + }, + { + "job": "whisper-cli_tiny.en_novad", + "wall": 15.358130356995389, + "inproc": null, + "rss_kb": 296748, + "user": 15.26, + "mean_runnable": 3.7, + "load1": 6.22, + "quiet": true, + "attempts": 2, + "round": 1 + }, + { + "job": "whisper-cli_tiny.en_vad", + "wall": 19.611542615108192, + "inproc": null, + "rss_kb": 314880, + "user": 18.22, + "mean_runnable": 3.7, + "load1": 7.3, + "quiet": true, + "attempts": 11, + "round": 1 + }, + { + "job": "onnxruntime_python_1thr_detect_only", + "wall": 1.241450318833813, + "inproc": 0.7934, + "rss_kb": 106540, + "user": 0.69, + "mean_runnable": 3.0, + "load1": 6.26, + "quiet": true, + "attempts": 1, + "round": 1 + }, + { + "job": "whisper-cli_tiny.en_vad", + "wall": 20.62023510481231, + "inproc": null, + "rss_kb": 314540, + "user": 18.96, + "mean_runnable": 5.4, + "load1": 4.6, + "quiet": true, + "attempts": 10, + "round": 2 + }, + { + "job": "onnxruntime_python_1thr_detect_only", + "wall": 0.7381441139150411, + "inproc": 0.607, + "rss_kb": 106712, + "user": 0.71, + "mean_runnable": 2.8, + "load1": 6.91, + "quiet": true, + "attempts": 4, + "round": 2 + }, + { + "job": "whisper-cli_tiny.en_novad", + "wall": 15.142986833816394, + "inproc": null, + "rss_kb": 296756, + "user": 15.04, + "mean_runnable": 2.9, + "load1": 8.26, + "quiet": true, + "attempts": 11, + "round": 2 + } +] \ No newline at end of file diff --git a/scripts/vad_bench/results/silero_speed_raw.json b/scripts/vad_bench/results/silero_speed_raw.json new file mode 100644 index 0000000..229868c --- /dev/null +++ b/scripts/vad_bench/results/silero_speed_raw.json @@ -0,0 +1,242 @@ +[ + { + "job": "wcpp_inproc_detect_only", + "wall": 0.44647544994950294, + "inproc": 0.4318, + "rss_kb": 36960, + "user": 0.43, + "mean_runnable": 2.5, + "load1": 14.27, + "quiet": true, + "attempts": 1, + "round": 0 + }, + { + "job": "wcpp_vad-speech-segments_cli", + "wall": 0.4505736359860748, + "inproc": null, + "rss_kb": 28800, + "user": 0.44, + "mean_runnable": 3.2, + "load1": 14.01, + "quiet": true, + "attempts": 1, + "round": 0 + }, + { + "job": "parakeet_cli_f32", + "wall": 0.4249067639466375, + "inproc": null, + "rss_kb": 61120, + "user": 0.4, + "mean_runnable": 1.8, + "load1": 13.79, + "quiet": true, + "attempts": 1, + "round": 0 + }, + { + "job": "parakeet_cli_f16", + "wall": 0.436342905042693, + "inproc": null, + "rss_kb": 61120, + "user": 0.41, + "mean_runnable": 2.9, + "load1": 13.44, + "quiet": true, + "attempts": 1, + "round": 0 + }, + { + "job": "wcpp_vad-speech-segments_cli", + "wall": 0.45197289693169296, + "inproc": null, + "rss_kb": 28640, + "user": 0.44, + "mean_runnable": 2.3, + "load1": 13.36, + "quiet": true, + "attempts": 1, + "round": 1 + }, + { + "job": "parakeet_cli_f32", + "wall": 0.4639807508792728, + "inproc": null, + "rss_kb": 61120, + "user": 0.44, + "mean_runnable": 2.7, + "load1": 13.29, + "quiet": true, + "attempts": 1, + "round": 1 + }, + { + "job": "parakeet_cli_f16", + "wall": 0.47317962581291795, + "inproc": null, + "rss_kb": 61120, + "user": 0.44, + "mean_runnable": 3.8, + "load1": 14.99, + "quiet": true, + "attempts": 2, + "round": 1 + }, + { + "job": "wcpp_inproc_detect_only", + "wall": 0.45590541488491, + "inproc": 0.4372, + "rss_kb": 36960, + "user": 0.44, + "mean_runnable": 3.2, + "load1": 15.93, + "quiet": true, + "attempts": 1, + "round": 1 + }, + { + "job": "parakeet_cli_f32", + "wall": 0.4558734449092299, + "inproc": null, + "rss_kb": 61120, + "user": 0.43, + "mean_runnable": 3.5, + "load1": 15.38, + "quiet": true, + "attempts": 2, + "round": 2 + }, + { + "job": "parakeet_cli_f16", + "wall": 0.5525356309954077, + "inproc": null, + "rss_kb": 61120, + "user": 0.47, + "mean_runnable": 3.8, + "load1": 13.82, + "quiet": true, + "attempts": 9, + "round": 2 + }, + { + "job": "wcpp_inproc_detect_only", + "wall": 0.46314642787911, + "inproc": 0.4413, + "rss_kb": 36800, + "user": 0.44, + "mean_runnable": 3.5, + "load1": 7.94, + "quiet": true, + "attempts": 5, + "round": 2 + }, + { + "job": "wcpp_vad-speech-segments_cli", + "wall": 0.45731914788484573, + "inproc": null, + "rss_kb": 28640, + "user": 0.44, + "mean_runnable": 2.6, + "load1": 7.26, + "quiet": true, + "attempts": 1, + "round": 2 + }, + { + "job": "parakeet_cli_f16", + "wall": 0.4371650768443942, + "inproc": null, + "rss_kb": 61120, + "user": 0.41, + "mean_runnable": 3.6, + "load1": 7.19, + "quiet": true, + "attempts": 4, + "round": 3 + }, + { + "job": "wcpp_inproc_detect_only", + "wall": 0.4506124691106379, + "inproc": 0.4376, + "rss_kb": 36960, + "user": 0.43, + "mean_runnable": 3.0, + "load1": 4.75, + "quiet": true, + "attempts": 3, + "round": 3 + }, + { + "job": "wcpp_vad-speech-segments_cli", + "wall": 0.4435456038918346, + "inproc": null, + "rss_kb": 28480, + "user": 0.43, + "mean_runnable": 2.6, + "load1": 4.4, + "quiet": true, + "attempts": 1, + "round": 3 + }, + { + "job": "parakeet_cli_f32", + "wall": 0.45499568711966276, + "inproc": null, + "rss_kb": 61120, + "user": 0.42, + "mean_runnable": 4.9, + "load1": 4.51, + "quiet": true, + "attempts": 1, + "round": 3 + }, + { + "job": "wcpp_inproc_detect_only", + "wall": 0.47262479993514717, + "inproc": 0.4548, + "rss_kb": 36800, + "user": 0.45, + "mean_runnable": 3.3, + "load1": 4.35, + "quiet": true, + "attempts": 1, + "round": 4 + }, + { + "job": "wcpp_vad-speech-segments_cli", + "wall": 0.45728860003873706, + "inproc": null, + "rss_kb": 28640, + "user": 0.44, + "mean_runnable": 4.2, + "load1": 4.29, + "quiet": true, + "attempts": 1, + "round": 4 + }, + { + "job": "parakeet_cli_f32", + "wall": 0.4525157401803881, + "inproc": null, + "rss_kb": 61120, + "user": 0.42, + "mean_runnable": 5.5, + "load1": 4.46, + "quiet": true, + "attempts": 1, + "round": 4 + }, + { + "job": "parakeet_cli_f16", + "wall": 0.45077538792975247, + "inproc": null, + "rss_kb": 61120, + "user": 0.42, + "mean_runnable": 4.1, + "load1": 4.16, + "quiet": true, + "attempts": 1, + "round": 4 + } +] \ No newline at end of file diff --git a/scripts/vad_bench/results/silero_versions_and_sha256.txt b/scripts/vad_bench/results/silero_versions_and_sha256.txt new file mode 100644 index 0000000..4156c5a --- /dev/null +++ b/scripts/vad_bench/results/silero_versions_and_sha256.txt @@ -0,0 +1,16 @@ +whisper.cpp 60c0be6ac8fa71b1a2ae2dd938a31a34a508e774 Fri Oct 2 15:21:13 2026 +0200 +set(WHISPER_VERSION_MAJOR 1) +parakeet.cpp c1a8391102d34074f48e758c441758327d2465b7 Merge origin/master into feat/vad-silero-api +parakeet ggml e705c5fe +gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 +cmake version 3.28.3 +8160489282352accc0e95925c2f6bf3d76fb8f7bccdce5c7e46808cb15e8443c silero-vad-f16.gguf +1398e5ce230bd8f20c06f825ed3a41dc4528ef7e6fe8f8ec8a5b7aedefbee943 silero-vad-f32.gguf +1a153a22f4509e292a94e67d6f9b85e8deb25b4988682b7e174c65279d8788e3 silero_vad.onnx +2aa269b785eeb53a82983a20501ddf7c1d9c48e33ab63a41391ac6c9f7fb6987 ggml-silero-v6.2.0.bin +abd79739d3ff4bc37fa1d24f1fca8244f2e31ec9f0bb7ea6244b185bfaa0b8e6 ggml-silero-v6.2.3-ggml.bin +921e4cf8686fdd993dcd081a5da5b6c365bfde1162e72b08d75ac75289920b1f ggml-tiny.en.bin +5b8e79aac593eb87aab62e2083d63e557570d7376b73e8f50c08adfbdbb69767 whisper.cpp/build/bin/whisper-cli +bf89b74f4a104ff32996eca405b5d5a13fefcdfb74882e5f131b849be37276ea whisper.cpp/build/bin/whisper-vad-speech-segments +31bc6b76f9929042e9b47bac13f521c732dc072818df81bee01531fcfc558aec wvad +ed5bb79c419fef6e8e31a3fee650fbdac2ed3c931f03102bbd6f5b7c04a0879e build/examples/cli/parakeet-cli diff --git a/scripts/vad_bench/results/slice_gate.log b/scripts/vad_bench/results/slice_gate.log new file mode 100644 index 0000000..a170abf --- /dev/null +++ b/scripts/vad_bench/results/slice_gate.log @@ -0,0 +1,496 @@ +22:25:57 load ultra-q8_0 full cold=False mean_runnable=22.2 load1=10.02 ok=False +22:26:37 load ultra-q8_0 full cold=False mean_runnable=8.2 load1=9.38 ok=False +22:27:17 load ultra-q8_0 full cold=False mean_runnable=6.0 load1=9.12 ok=False +22:27:57 load ultra-q8_0 full cold=False mean_runnable=13.8 load1=13.77 ok=False +22:28:37 load ultra-q8_0 full cold=False mean_runnable=8.9 load1=11.14 ok=False +22:29:17 load ultra-q8_0 full cold=False mean_runnable=16.6 load1=13.7 ok=False +22:29:57 load ultra-q8_0 full cold=False mean_runnable=5.2 load1=9.73 ok=False +22:30:37 load ultra-q8_0 full cold=False mean_runnable=3.8 load1=6.89 ok=True +22:30:50 load ultra-q8_0 full cold=True mean_runnable=3.3 load1=6.62 ok=True +22:31:05 load ultra-q8_0 slice cold=False mean_runnable=2.6 load1=5.83 ok=True +22:31:15 load ultra-q8_0 slice cold=True mean_runnable=4.2 load1=5.47 ok=False +22:31:55 load ultra-q8_0 slice cold=True mean_runnable=4.5 load1=5.48 ok=False +22:32:35 load ultra-q8_0 slice cold=True mean_runnable=8.7 load1=6.49 ok=False +22:33:15 load ultra-q8_0 slice cold=True mean_runnable=10.5 load1=11.38 ok=False +22:33:55 load ultra-q8_0 slice cold=True mean_runnable=5.6 load1=10.29 ok=False +22:34:35 load ultra-q8_0 slice cold=True mean_runnable=2.1 load1=6.89 ok=True +22:34:45 load ultra-f16 full cold=False mean_runnable=4.1 load1=6.14 ok=False +22:35:25 load ultra-f16 full cold=False mean_runnable=3.2 load1=4.56 ok=True +22:35:40 load ultra-f16 full cold=True mean_runnable=2.5 load1=4.13 ok=True +22:35:57 load ultra-f16 slice cold=False mean_runnable=2.7 load1=3.82 ok=True +22:36:07 load ultra-f16 slice cold=True mean_runnable=2.7 load1=3.46 ok=True +22:36:17 load redux-keep full cold=False mean_runnable=2.3 load1=3.32 ok=True +22:36:28 load redux-keep full cold=True mean_runnable=2.2 load1=3.12 ok=True +22:36:40 load redux-keep slice cold=False mean_runnable=2.5 load1=2.94 ok=True +22:36:50 load redux-keep slice cold=True mean_runnable=2.3 load1=2.72 ok=True +22:37:00 load redux-deq-f16 full cold=False mean_runnable=1.9 load1=2.53 ok=True +22:37:15 load redux-deq-f16 full cold=True mean_runnable=2.1 load1=2.63 ok=True +22:37:32 load redux-deq-f16 slice cold=False mean_runnable=2.8 load1=3.61 ok=True +22:37:42 load redux-deq-f16 slice cold=True mean_runnable=2.6 load1=3.45 ok=True +22:37:52 load silero-vad-f16 cold=False mean_runnable=1.6 load1=3.22 ok=True +22:38:02 load silero-vad-f16 cold=True mean_runnable=2.5 load1=3.04 ok=True +22:38:26 load ultra-q8_0 full cold=False mean_runnable=11.6 load1=5.64 ok=False +22:39:06 load ultra-q8_0 full cold=False mean_runnable=9.6 load1=9.64 ok=False +22:39:46 load ultra-q8_0 full cold=False mean_runnable=3.4 load1=7.5 ok=True +22:39:59 load ultra-q8_0 full cold=True mean_runnable=5.5 load1=9.37 ok=False +22:40:39 load ultra-q8_0 full cold=True mean_runnable=17.2 load1=9.03 ok=False +22:41:19 load ultra-q8_0 full cold=True mean_runnable=27.4 load1=19.5 ok=False +22:41:59 load ultra-q8_0 full cold=True mean_runnable=8.2 load1=17.44 ok=False +22:42:39 load ultra-q8_0 full cold=True mean_runnable=22.4 load1=20.98 ok=False +22:43:19 load ultra-q8_0 full cold=True mean_runnable=14.9 load1=18.05 ok=False +22:43:59 load ultra-q8_0 full cold=True mean_runnable=4.2 load1=13.28 ok=False +22:44:39 load ultra-q8_0 full cold=True mean_runnable=8.7 load1=9.15 ok=False +22:45:19 load ultra-q8_0 full cold=True mean_runnable=3.4 load1=7.49 ok=True +22:45:34 load ultra-q8_0 slice cold=False mean_runnable=4.7 load1=6.71 ok=False +22:46:14 load ultra-q8_0 slice cold=False mean_runnable=10.6 load1=8.53 ok=False +22:46:54 load ultra-q8_0 slice cold=False mean_runnable=9.9 load1=10.03 ok=False +22:47:34 load ultra-q8_0 slice cold=False mean_runnable=4.1 load1=8.36 ok=False +22:48:14 load ultra-q8_0 slice cold=False mean_runnable=17.4 load1=10.4 ok=False +22:48:54 load ultra-q8_0 slice cold=False mean_runnable=17.1 load1=11.38 ok=False +22:49:34 load ultra-q8_0 slice cold=False mean_runnable=4.3 load1=10.02 ok=False +22:50:14 load ultra-q8_0 slice cold=False mean_runnable=3.6 load1=7.13 ok=True +22:50:24 load ultra-q8_0 slice cold=True mean_runnable=6.3 load1=6.65 ok=False +22:51:04 load ultra-q8_0 slice cold=True mean_runnable=6.8 load1=5.5 ok=False +22:51:44 load ultra-q8_0 slice cold=True mean_runnable=35.9 load1=9.91 ok=False +22:52:24 load ultra-q8_0 slice cold=True mean_runnable=50.3 load1=21.91 ok=False +22:53:04 load ultra-q8_0 slice cold=True mean_runnable=29.2 load1=30.68 ok=False +22:53:44 load ultra-q8_0 slice cold=True mean_runnable=50.9 load1=35.89 ok=False +22:54:24 load ultra-q8_0 slice cold=True mean_runnable=54.1 load1=45.19 ok=False +22:55:04 load ultra-q8_0 slice cold=True mean_runnable=15.3 load1=40.41 ok=False +22:55:44 load ultra-q8_0 slice cold=True mean_runnable=57.2 load1=44.67 ok=False +22:56:24 load ultra-q8_0 slice cold=True mean_runnable=56.6 load1=54.46 ok=False +22:57:05 load ultra-q8_0 slice cold=True mean_runnable=62.5 load1=60.04 ok=False +22:57:45 load ultra-q8_0 slice cold=True mean_runnable=52.5 load1=61.9 ok=False +22:58:39 speed round0 t1 mean_runnable=44.2 load1=60.45 ok=False +not quiet, retry in 30s +22:59:19 speed round0 t1 mean_runnable=33.6 load1=52.13 ok=False +not quiet, retry in 30s +22:59:59 speed round0 t1 mean_runnable=14.6 load1=38.61 ok=False +not quiet, retry in 30s +23:00:39 speed round0 t1 mean_runnable=21.9 load1=32.01 ok=False +not quiet, retry in 30s +23:01:19 speed round0 t1 mean_runnable=22.2 load1=27.61 ok=False +not quiet, retry in 30s +23:01:59 speed round0 t1 mean_runnable=52.0 load1=46.33 ok=False +not quiet, retry in 30s +23:02:39 speed round0 t1 mean_runnable=23.1 load1=46.53 ok=False +not quiet, retry in 30s +23:03:19 speed round0 t1 mean_runnable=24.4 load1=41.51 ok=False +not quiet, retry in 30s +23:03:59 speed round0 t1 mean_runnable=18.2 load1=33.9 ok=False +not quiet, retry in 30s +23:04:39 speed round0 t1 mean_runnable=16.6 load1=29.14 ok=False +not quiet, retry in 30s +23:05:20 speed round0 t1 mean_runnable=65.3 load1=32.34 ok=False +not quiet, retry in 30s +23:06:00 speed round0 t1 mean_runnable=50.5 load1=44.47 ok=False +not quiet, retry in 30s +23:06:40 speed round0 t1 mean_runnable=31.5 load1=48.52 ok=False +not quiet, retry in 30s +23:07:20 speed round0 t1 mean_runnable=24.8 load1=36.49 ok=False +not quiet, retry in 30s +23:08:00 speed round0 t1 mean_runnable=20.0 load1=32.66 ok=False +not quiet, retry in 30s +23:08:40 speed round0 t1 mean_runnable=6.3 load1=19.66 ok=False +not quiet, retry in 30s +23:09:20 speed round0 t1 mean_runnable=9.8 load1=13.76 ok=False +not quiet, retry in 30s +23:10:00 speed round0 t1 mean_runnable=11.1 load1=13.02 ok=False +not quiet, retry in 30s +23:10:40 speed round0 t1 mean_runnable=4.7 load1=9.44 ok=False +not quiet, retry in 30s +23:11:20 speed round0 t1 mean_runnable=8.3 load1=8.63 ok=False +not quiet, retry in 30s +23:12:00 speed round0 t1 mean_runnable=8.2 load1=13.15 ok=False +not quiet, retry in 30s +23:12:40 speed round0 t1 mean_runnable=11.8 load1=12.36 ok=False +not quiet, retry in 30s +23:13:20 speed round0 t1 mean_runnable=22.9 load1=14.04 ok=False +not quiet, retry in 30s +23:14:00 speed round0 t1 mean_runnable=6.2 load1=13.68 ok=False +not quiet, retry in 30s +23:14:40 speed round0 t1 mean_runnable=3.9 load1=9.33 ok=True +23:14:51 speed round0 t8 mean_runnable=4.2 load1=7.71 ok=False +not quiet, retry in 30s +23:15:31 speed round0 t8 mean_runnable=4.9 load1=5.74 ok=False +not quiet, retry in 30s +23:16:11 speed round0 t8 mean_runnable=4.0 load1=5.63 ok=False +not quiet, retry in 30s +23:16:51 speed round0 t8 mean_runnable=7.0 load1=6.84 ok=False +not quiet, retry in 30s +23:17:31 speed round0 t8 mean_runnable=4.7 load1=5.52 ok=False +not quiet, retry in 30s +23:18:11 speed round0 t8 mean_runnable=5.7 load1=5.22 ok=False +not quiet, retry in 30s +23:18:51 speed round0 t8 mean_runnable=4.8 load1=5.23 ok=False +not quiet, retry in 30s +23:19:31 speed round0 t8 mean_runnable=8.6 load1=6.58 ok=False +not quiet, retry in 30s +23:20:11 speed round0 t8 mean_runnable=4.0 load1=5.76 ok=True +23:20:23 speed round1 t1 mean_runnable=3.5 load1=5.41 ok=True +23:20:34 speed round1 t8 mean_runnable=4.8 load1=5.66 ok=False +not quiet, retry in 30s +23:21:14 speed round1 t8 mean_runnable=15.0 load1=9.33 ok=False +not quiet, retry in 30s +23:21:54 speed round1 t8 mean_runnable=4.0 load1=15.65 ok=True +23:22:05 speed round2 t1 mean_runnable=14.3 load1=17.87 ok=False +not quiet, retry in 30s +23:22:45 speed round2 t1 mean_runnable=62.0 load1=26.47 ok=False +not quiet, retry in 30s +23:23:26 speed round2 t1 mean_runnable=71.2 load1=45.48 ok=False +not quiet, retry in 30s +23:24:06 speed round2 t1 mean_runnable=11.1 load1=29.91 ok=False +not quiet, retry in 30s +23:24:46 speed round2 t1 mean_runnable=2.4 load1=17.33 ok=True +23:24:57 speed round2 t8 mean_runnable=2.8 load1=14.43 ok=True +23:25:08 speed round3 t1 mean_runnable=3.6 load1=12.66 ok=True +23:25:20 speed round3 t8 mean_runnable=2.4 load1=11.1 ok=True +23:25:31 speed round4 t1 mean_runnable=3.1 load1=9.3 ok=True +23:25:42 speed round4 t8 mean_runnable=2.4 load1=8.26 ok=True +--- rerun speed with local data copy +23:26:32 speed round0 t1 mean_runnable=3.0 load1=5.14 ok=True +23:27:37 speed round0 t8 mean_runnable=2.2 load1=5.01 ok=True +23:28:09 speed round1 t1 mean_runnable=3.2 load1=5.26 ok=True +23:29:16 speed round1 t8 mean_runnable=11.8 load1=7.37 ok=False +not quiet, retry in 30s +23:29:56 speed round1 t8 mean_runnable=19.1 load1=12.11 ok=False +not quiet, retry in 30s +23:30:36 speed round1 t8 mean_runnable=3.4 load1=13.89 ok=True +23:31:09 speed round2 t1 mean_runnable=3.5 load1=12.67 ok=True +23:32:17 speed round2 t8 mean_runnable=37.2 load1=22.05 ok=False +not quiet, retry in 30s +23:32:57 speed round2 t8 mean_runnable=7.2 load1=19.81 ok=False +not quiet, retry in 30s +23:33:37 speed round2 t8 mean_runnable=39.3 load1=23.89 ok=False +not quiet, retry in 30s +23:34:17 speed round2 t8 mean_runnable=5.8 load1=24.75 ok=False +not quiet, retry in 30s +23:34:57 speed round2 t8 mean_runnable=10.6 load1=16.44 ok=False +not quiet, retry in 30s +23:35:37 speed round2 t8 mean_runnable=17.6 load1=19.24 ok=False +not quiet, retry in 30s +23:36:17 speed round2 t8 mean_runnable=3.8 load1=12.7 ok=True +23:36:51 speed round3 t1 mean_runnable=3.7 load1=11.9 ok=True +23:37:58 speed round3 t8 mean_runnable=8.5 load1=9.58 ok=False +not quiet, retry in 30s +23:38:38 speed round3 t8 mean_runnable=9.4 load1=8.78 ok=False +not quiet, retry in 30s +23:39:18 speed round3 t8 mean_runnable=4.2 load1=8.13 ok=False +not quiet, retry in 30s +23:39:58 speed round3 t8 mean_runnable=7.7 load1=7.36 ok=False +not quiet, retry in 30s +23:40:38 speed round3 t8 mean_runnable=11.0 load1=6.69 ok=False +not quiet, retry in 30s +23:41:18 speed round3 t8 mean_runnable=5.0 load1=7.02 ok=False +not quiet, retry in 30s +23:41:58 speed round3 t8 mean_runnable=22.2 load1=13.86 ok=False +not quiet, retry in 30s +23:42:38 speed round3 t8 mean_runnable=9.4 load1=13.08 ok=False +not quiet, retry in 30s +23:43:18 speed round3 t8 mean_runnable=12.7 load1=13.06 ok=False +not quiet, retry in 30s +23:43:58 speed round3 t8 mean_runnable=7.4 load1=12.51 ok=False +not quiet, retry in 30s +23:44:38 speed round3 t8 mean_runnable=12.9 load1=12.55 ok=False +not quiet, retry in 30s +23:45:18 speed round3 t8 mean_runnable=10.8 load1=11.65 ok=False +not quiet, retry in 30s +23:45:58 speed round3 t8 mean_runnable=17.6 load1=14.55 ok=False +not quiet, retry in 30s +23:46:38 speed round3 t8 mean_runnable=30.9 load1=22.36 ok=False +not quiet, retry in 30s +23:47:18 speed round3 t8 mean_runnable=23.1 load1=25.02 ok=False +not quiet, retry in 30s +23:47:58 speed round3 t8 mean_runnable=83.7 load1=43.53 ok=False +not quiet, retry in 30s +23:48:38 speed round3 t8 mean_runnable=119.0 load1=67.52 ok=False +not quiet, retry in 30s +23:49:18 speed round3 t8 mean_runnable=90.1 load1=103.67 ok=False +not quiet, retry in 30s +23:49:58 speed round3 t8 mean_runnable=113.4 load1=96.99 ok=False +not quiet, retry in 30s +23:50:38 speed round3 t8 mean_runnable=119.6 load1=119.77 ok=False +not quiet, retry in 30s +23:51:18 speed round3 t8 mean_runnable=53.0 load1=102.39 ok=False +not quiet, retry in 30s +23:51:58 speed round3 t8 mean_runnable=61.3 load1=87.49 ok=False +not quiet, retry in 30s +23:52:38 speed round3 t8 mean_runnable=51.3 load1=88.8 ok=False +not quiet, retry in 30s +23:53:18 speed round3 t8 mean_runnable=72.8 load1=88.13 ok=False +not quiet, retry in 30s +23:53:58 speed round3 t8 mean_runnable=65.3 load1=83.06 ok=False +not quiet, retry in 30s +23:54:38 speed round3 t8 mean_runnable=80.5 load1=82.42 ok=False +not quiet, retry in 30s +23:55:18 speed round3 t8 mean_runnable=76.4 load1=89.03 ok=False +not quiet, retry in 30s +23:55:58 speed round3 t8 mean_runnable=46.8 load1=80.18 ok=False +not quiet, retry in 30s +23:56:38 speed round3 t8 mean_runnable=14.1 load1=61.31 ok=False +not quiet, retry in 30s +23:57:18 speed round3 t8 mean_runnable=18.6 load1=42.61 ok=False +not quiet, retry in 30s +23:57:58 speed round3 t8 mean_runnable=24.1 load1=31.22 ok=False +not quiet, retry in 30s +23:58:38 speed round3 t8 mean_runnable=30.4 load1=27.78 ok=False +not quiet, retry in 30s +23:59:18 speed round3 t8 mean_runnable=83.2 load1=52.49 ok=False +not quiet, retry in 30s +23:59:58 speed round3 t8 mean_runnable=41.8 load1=48.26 ok=False +not quiet, retry in 30s +00:00:38 speed round3 t8 mean_runnable=72.8 load1=55.97 ok=False +not quiet, retry in 30s +00:01:18 speed round3 t8 mean_runnable=10.9 load1=46.46 ok=False +not quiet, retry in 30s +00:01:58 speed round3 t8 mean_runnable=16.4 load1=31.58 ok=False +not quiet, retry in 30s +00:02:38 speed round3 t8 mean_runnable=18.3 load1=25.8 ok=False +not quiet, retry in 30s +00:03:18 speed round3 t8 mean_runnable=61.1 load1=26.92 ok=False +not quiet, retry in 30s +00:03:58 speed round3 t8 mean_runnable=108.9 load1=65.61 ok=False +not quiet, retry in 30s +00:04:38 speed round3 t8 mean_runnable=78.7 load1=66.31 ok=False +not quiet, retry in 30s +00:05:18 speed round3 t8 mean_runnable=40.3 load1=64.74 ok=False +not quiet, retry in 30s +00:05:58 speed round3 t8 mean_runnable=32.8 load1=52.31 ok=False +not quiet, retry in 30s +00:06:38 speed round3 t8 mean_runnable=56.0 load1=49.43 ok=False +not quiet, retry in 30s +00:07:18 speed round3 t8 mean_runnable=60.2 load1=60.68 ok=False +not quiet, retry in 30s +00:07:58 speed round3 t8 mean_runnable=113.5 load1=81.87 ok=False +not quiet, retry in 30s +00:08:38 speed round3 t8 mean_runnable=64.5 load1=80.12 ok=False +not quiet, retry in 30s +00:09:18 speed round3 t8 mean_runnable=52.6 load1=71.99 ok=False +not quiet, retry in 30s +00:09:58 speed round3 t8 mean_runnable=51.6 load1=65.16 ok=False +not quiet, retry in 30s +00:10:38 speed round3 t8 mean_runnable=79.2 load1=72.1 ok=False +not quiet, retry in 30s +00:11:18 speed round3 t8 mean_runnable=47.8 load1=68.7 ok=False +not quiet, retry in 30s +00:11:58 speed round3 t8 mean_runnable=29.4 load1=54.79 ok=False +not quiet, retry in 30s +00:12:38 speed round3 t8 mean_runnable=36.2 load1=45.03 ok=False +not quiet, retry in 30s +00:13:18 speed round3 t8 mean_runnable=46.0 load1=50.37 ok=False +not quiet, retry in 30s +00:13:58 speed round3 t8 mean_runnable=15.9 load1=37.31 ok=False +not quiet, retry in 30s +00:14:38 speed round3 t8 mean_runnable=12.8 load1=24.53 ok=False +not quiet, retry in 30s +00:15:18 speed round3 t8 mean_runnable=9.1 load1=20.61 ok=False +not quiet, retry in 30s +00:15:58 speed round3 t8 mean_runnable=26.6 load1=23.46 ok=False +not quiet, retry in 30s +00:16:38 speed round3 t8 mean_runnable=44.2 load1=29.85 ok=False +not quiet, retry in 30s +00:17:18 speed round3 t8 mean_runnable=20.3 load1=22.12 ok=False +not quiet, retry in 30s +00:17:58 speed round3 t8 mean_runnable=10.2 load1=23.32 ok=False +not quiet, retry in 30s +00:18:38 speed round3 t8 mean_runnable=20.4 load1=19.81 ok=False +not quiet, retry in 30s +00:19:18 speed round3 t8 mean_runnable=33.9 load1=18.64 ok=False +not quiet, retry in 30s +00:19:58 speed round3 t8 mean_runnable=38.0 load1=33.8 ok=False +not quiet, retry in 30s +00:20:38 speed round3 t8 mean_runnable=36.8 load1=34.77 ok=False +not quiet, retry in 30s +00:21:18 speed round3 t8 mean_runnable=29.9 load1=38.84 ok=False +not quiet, retry in 30s +00:21:58 speed round3 t8 mean_runnable=34.6 load1=41.54 ok=False +not quiet, retry in 30s +00:22:38 speed round3 t8 mean_runnable=27.6 load1=36.55 ok=False +not quiet, retry in 30s +00:23:18 speed round3 t8 mean_runnable=13.7 load1=27.11 ok=False +not quiet, retry in 30s +00:23:58 speed round3 t8 mean_runnable=37.2 load1=35.28 ok=False +not quiet, retry in 30s +00:24:38 speed round3 t8 mean_runnable=29.1 load1=48.33 ok=False +not quiet, retry in 30s +00:25:18 speed round3 t8 mean_runnable=5.2 load1=32.66 ok=False +not quiet, retry in 30s +00:25:58 speed round3 t8 mean_runnable=5.4 load1=21.01 ok=False +not quiet, retry in 30s +00:26:38 speed round3 t8 mean_runnable=63.4 load1=20.82 ok=False +not quiet, retry in 30s +00:27:18 speed round3 t8 mean_runnable=98.2 load1=35.79 ok=False +not quiet, retry in 30s +00:27:58 speed round3 t8 mean_runnable=10.6 load1=28.54 ok=False +not quiet, retry in 30s +00:28:38 speed round3 t8 mean_runnable=26.8 load1=40.58 ok=False +not quiet, retry in 30s +00:29:18 speed round3 t8 mean_runnable=93.5 load1=58.46 ok=False +not quiet, retry in 30s +00:29:58 speed round3 t8 mean_runnable=11.2 load1=53.45 ok=False +not quiet, retry in 30s +00:30:38 speed round3 t8 mean_runnable=35.5 load1=37.22 ok=False +not quiet, retry in 30s +00:31:18 speed round3 t8 mean_runnable=92.5 load1=57.18 ok=False +not quiet, retry in 30s +00:31:58 speed round3 t8 mean_runnable=89.3 load1=72.14 ok=False +not quiet, retry in 30s +00:32:38 speed round3 t8 mean_runnable=10.2 load1=50.99 ok=False +not quiet, retry in 30s +00:33:18 speed round3 t8 mean_runnable=12.2 load1=37.38 ok=False +not quiet, retry in 30s +00:33:58 speed round3 t8 mean_runnable=15.9 load1=26.78 ok=False +not quiet, retry in 30s +00:34:38 speed round3 t8 mean_runnable=21.4 load1=23.94 ok=False +not quiet, retry in 30s +00:35:18 speed round3 t8 mean_runnable=17.2 load1=23.2 ok=False +not quiet, retry in 30s +00:35:58 speed round3 t8 mean_runnable=51.4 load1=32.42 ok=False +not quiet, retry in 30s +00:36:38 speed round3 t8 mean_runnable=8.5 load1=29.3 ok=False +not quiet, retry in 30s +00:37:18 speed round3 t8 mean_runnable=23.8 load1=23.32 ok=False +not quiet, retry in 30s +00:37:58 speed round3 t8 mean_runnable=58.1 load1=31.52 ok=False +not quiet, retry in 30s +00:38:38 speed round3 t8 mean_runnable=67.2 load1=47.1 ok=False +not quiet, retry in 30s +00:39:18 speed round3 t8 mean_runnable=19.9 load1=45.81 ok=False +not quiet, retry in 30s +00:39:58 speed round3 t8 mean_runnable=27.9 load1=39.97 ok=False +not quiet, retry in 30s +00:40:38 speed round3 t8 mean_runnable=23.8 load1=31.28 ok=False +not quiet, retry in 30s +00:41:18 speed round3 t8 mean_runnable=30.2 load1=22.59 ok=False +not quiet, retry in 30s +00:41:58 speed round3 t8 mean_runnable=70.0 load1=40.36 ok=False +not quiet, retry in 30s +00:42:38 speed round3 t8 mean_runnable=61.5 load1=54.21 ok=False +not quiet, retry in 30s +00:43:18 speed round3 t8 mean_runnable=39.9 load1=59.64 ok=False +not quiet, retry in 30s +00:43:58 speed round3 t8 mean_runnable=50.9 load1=52.32 ok=False +not quiet, retry in 30s +00:44:38 speed round3 t8 mean_runnable=35.5 load1=48.34 ok=False +not quiet, retry in 30s +00:45:18 speed round3 t8 mean_runnable=11.1 load1=34.3 ok=False +not quiet, retry in 30s +00:45:58 speed round3 t8 mean_runnable=8.1 load1=22.41 ok=False +not quiet, retry in 30s +00:46:39 speed round3 t8 mean_runnable=89.8 load1=34.07 ok=False +not quiet, retry in 30s +00:47:19 speed round3 t8 mean_runnable=21.6 load1=27.81 ok=False +not quiet, retry in 30s +00:47:59 speed round3 t8 mean_runnable=55.2 load1=45.27 ok=False +not quiet, retry in 30s +00:48:39 speed round3 t8 mean_runnable=53.6 load1=62.55 ok=False +not quiet, retry in 30s +00:49:19 speed round3 t8 mean_runnable=90.2 load1=67.05 ok=False +not quiet, retry in 30s +00:49:59 speed round3 t8 mean_runnable=42.9 load1=56.68 ok=False +not quiet, retry in 30s +00:50:39 speed round3 t8 mean_runnable=69.0 load1=55.88 ok=False +not quiet, retry in 30s +00:51:19 speed round3 t8 mean_runnable=24.2 load1=39.1 ok=False +not quiet, retry in 30s +00:51:59 speed round3 t8 mean_runnable=4.6 load1=22.81 ok=False +not quiet, retry in 30s +00:52:39 speed round3 t8 mean_runnable=10.3 load1=13.88 ok=False +not quiet, retry in 30s +00:53:19 speed round3 t8 mean_runnable=5.5 load1=9.36 ok=False +not quiet, retry in 30s +00:53:59 speed round3 t8 mean_runnable=5.2 load1=6.94 ok=False +not quiet, retry in 30s +00:54:39 speed round3 t8 mean_runnable=11.3 load1=6.98 ok=False +not quiet, retry in 30s +00:55:19 speed round3 t8 mean_runnable=13.4 load1=9.84 ok=False +not quiet, retry in 30s +00:55:59 speed round3 t8 mean_runnable=3.7 load1=9.3 ok=True +00:56:34 speed round4 t1 mean_runnable=32.6 load1=18.42 ok=False +not quiet, retry in 30s +00:57:14 speed round4 t1 mean_runnable=30.8 load1=28.52 ok=False +not quiet, retry in 30s +00:57:54 speed round4 t1 mean_runnable=10.0 load1=33.09 ok=False +not quiet, retry in 30s +00:58:34 speed round4 t1 mean_runnable=7.3 load1=20.45 ok=False +not quiet, retry in 30s +00:59:14 speed round4 t1 mean_runnable=36.8 load1=24.39 ok=False +not quiet, retry in 30s +00:59:54 speed round4 t1 mean_runnable=17.9 load1=17.4 ok=False +not quiet, retry in 30s +01:00:34 speed round4 t1 mean_runnable=67.2 load1=21.99 ok=False +not quiet, retry in 30s +01:01:14 speed round4 t1 mean_runnable=8.2 load1=26.74 ok=False +not quiet, retry in 30s +01:01:54 speed round4 t1 mean_runnable=22.9 load1=20.96 ok=False +not quiet, retry in 30s +01:02:34 speed round4 t1 mean_runnable=6.0 load1=13.64 ok=False +not quiet, retry in 30s +01:03:14 speed round4 t1 mean_runnable=4.6 load1=11.96 ok=False +not quiet, retry in 30s +01:03:54 speed round4 t1 mean_runnable=10.8 load1=9.54 ok=False +not quiet, retry in 30s +01:04:34 speed round4 t1 mean_runnable=24.8 load1=22.15 ok=False +not quiet, retry in 30s +01:05:14 speed round4 t1 mean_runnable=105.2 load1=43.74 ok=False +not quiet, retry in 30s +01:05:54 speed round4 t1 mean_runnable=8.0 load1=40.64 ok=False +not quiet, retry in 30s +01:06:34 speed round4 t1 mean_runnable=8.0 load1=26.94 ok=False +not quiet, retry in 30s +01:07:14 speed round4 t1 mean_runnable=8.7 load1=16.64 ok=False +not quiet, retry in 30s +01:07:54 speed round4 t1 mean_runnable=13.6 load1=13.63 ok=False +not quiet, retry in 30s +01:08:34 speed round4 t1 mean_runnable=23.4 load1=14.55 ok=False +not quiet, retry in 30s +01:09:15 speed round4 t1 mean_runnable=23.7 load1=23.17 ok=False +not quiet, retry in 30s +01:09:55 speed round4 t1 mean_runnable=21.6 load1=25.05 ok=False +not quiet, retry in 30s +01:10:35 speed round4 t1 mean_runnable=41.0 load1=28.11 ok=False +not quiet, retry in 30s +01:11:15 speed round4 t1 mean_runnable=10.7 load1=20.96 ok=False +not quiet, retry in 30s +01:11:55 speed round4 t1 mean_runnable=5.5 load1=18.81 ok=False +not quiet, retry in 30s +01:12:35 speed round4 t1 mean_runnable=5.8 load1=13.02 ok=False +not quiet, retry in 30s +01:13:15 speed round4 t1 mean_runnable=7.2 load1=9.47 ok=False +not quiet, retry in 30s +01:13:55 speed round4 t1 mean_runnable=14.8 load1=8.18 ok=False +not quiet, retry in 30s +01:14:35 speed round4 t1 mean_runnable=56.4 load1=18.69 ok=False +not quiet, retry in 30s +01:15:15 speed round4 t1 mean_runnable=122.5 load1=58.63 ok=False +not quiet, retry in 30s +01:15:55 speed round4 t1 mean_runnable=17.6 load1=46.06 ok=False +not quiet, retry in 30s +01:16:35 speed round4 t1 mean_runnable=14.4 load1=46.05 ok=False +not quiet, retry in 30s +01:17:15 speed round4 t1 mean_runnable=26.8 load1=37.02 ok=False +not quiet, retry in 30s +01:17:55 speed round4 t1 mean_runnable=9.6 load1=23.92 ok=False +not quiet, retry in 30s +01:18:35 speed round4 t1 mean_runnable=4.5 load1=16.83 ok=False +not quiet, retry in 30s +01:19:15 speed round4 t1 mean_runnable=6.3 load1=11.49 ok=False +not quiet, retry in 30s +01:19:55 speed round4 t1 mean_runnable=18.8 load1=17.0 ok=False +not quiet, retry in 30s +01:20:35 speed round4 t1 mean_runnable=9.4 load1=20.12 ok=False +not quiet, retry in 30s +01:21:15 speed round4 t1 mean_runnable=9.3 load1=14.33 ok=False +not quiet, retry in 30s +01:21:55 speed round4 t1 mean_runnable=6.3 load1=10.56 ok=False +not quiet, retry in 30s +01:22:35 speed round4 t1 mean_runnable=5.5 load1=7.38 ok=False +not quiet, retry in 30s +01:23:15 speed round4 t1 mean_runnable=5.5 load1=6.77 ok=False +not quiet, retry in 30s +01:23:55 speed round4 t1 mean_runnable=4.2 load1=6.94 ok=False +not quiet, retry in 30s +01:24:35 speed round4 t1 mean_runnable=4.7 load1=5.15 ok=False +not quiet, retry in 30s +01:25:15 speed round4 t1 mean_runnable=3.2 load1=4.54 ok=True +01:26:20 speed round4 t8 mean_runnable=8.2 load1=4.15 ok=True diff --git a/scripts/vad_bench/results/slice_load.json b/scripts/vad_bench/results/slice_load.json new file mode 100644 index 0000000..a157216 --- /dev/null +++ b/scripts/vad_bench/results/slice_load.json @@ -0,0 +1,218 @@ +{ + "ultra-q8_0 full|warm": [ + 271.7, + 269.9, + 280.0, + 269.6, + 270.3, + 277.3, + 274.7, + 277.0, + 300.8, + 279.3 + ], + "ultra-q8_0 full|cold": [ + 451.1, + 440.4, + 460.9, + 427.3, + 423.8, + 419.5, + 420.4, + 416.2, + 419.5, + 424.3 + ], + "ultra-q8_0 slice|warm": [ + 1.4, + 0.5, + 0.4, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2, + 0.2 + ], + "ultra-q8_0 slice|cold": [ + 5.1, + 2.9, + 2.3, + 2.2, + 1.9, + 2.1, + 2.1, + 2.0, + 2.1, + 2.0 + ], + "ultra-f16 full|warm": [ + 430.2, + 428.6, + 425.2, + 425.8, + 426.9, + 430.0, + 435.2, + 435.0, + 439.2, + 426.8 + ], + "ultra-f16 full|cold": [ + 678.2, + 684.3, + 664.1, + 666.3, + 660.9, + 672.9, + 664.9, + 658.9, + 674.0, + 664.7 + ], + "ultra-f16 slice|warm": [ + 2.2, + 0.8, + 0.6, + 0.5, + 0.4, + 0.4, + 0.3, + 0.3, + 0.4, + 0.3 + ], + "ultra-f16 slice|cold": [ + 6.7, + 4.2, + 3.3, + 3.6, + 3.6, + 3.2, + 3.2, + 3.4, + 3.6, + 3.4 + ], + "redux-keep full|warm": [ + 94.2, + 88.1, + 86.7, + 90.4, + 93.3, + 93.3, + 91.8, + 92.1, + 90.3, + 93.0 + ], + "redux-keep full|cold": [ + 145.0, + 130.6, + 129.9, + 135.2, + 122.3, + 126.0, + 128.3, + 128.1, + 122.2, + 125.7 + ], + "redux-keep slice|warm": [ + 2.2, + 0.8, + 0.6, + 0.5, + 0.4, + 0.4, + 0.4, + 0.4, + 0.5, + 0.4 + ], + "redux-keep slice|cold": [ + 6.9, + 4.4, + 3.5, + 3.7, + 3.7, + 3.6, + 3.9, + 4.1, + 3.7, + 3.5 + ], + "redux-deq-f16 full|warm": [ + 439.0, + 442.8, + 443.6, + 432.5, + 429.8, + 434.9, + 432.5, + 436.7, + 433.8, + 425.7 + ], + "redux-deq-f16 full|cold": [ + 659.3, + 682.1, + 665.0, + 677.6, + 667.6, + 666.2, + 659.4, + 686.0, + 706.4, + 670.0 + ], + "redux-deq-f16 slice|warm": [ + 1.7, + 0.5, + 0.3, + 0.3, + 0.2, + 0.2, + 0.2, + 0.3, + 0.2, + 0.2 + ], + "redux-deq-f16 slice|cold": [ + 6.7, + 4.6, + 3.8, + 3.8, + 3.8, + 3.7, + 3.7, + 3.8, + 3.7, + 3.9 + ], + "silero-vad-f16|warm": [ + 0.9, + 0.5, + 0.4, + 0.4, + 0.4, + 0.4, + 0.4, + 0.4, + 0.4, + 0.4 + ], + "silero-vad-f16|cold": [ + 3.8, + 2.6, + 1.5, + 1.5, + 1.5, + 1.5, + 1.5, + 1.6, + 1.6, + 1.5 + ] +} \ No newline at end of file diff --git a/scripts/vad_bench/results/slice_rss.txt b/scripts/vad_bench/results/slice_rss.txt new file mode 100644 index 0000000..6afa114 --- /dev/null +++ b/scripts/vad_bench/results/slice_rss.txt @@ -0,0 +1,18 @@ +ultra-q8_0 full load-only peak RSS 905.1 MiB +ultra-q8_0 full load+vad33s peak RSS 1133.4 MiB +ultra-q8_0 slice load-only peak RSS 11.7 MiB +ultra-q8_0 slice load+vad33s peak RSS 240.7 MiB +ultra-f16 full load-only peak RSS 1382.4 MiB +ultra-f16 full load+vad33s peak RSS 1612.3 MiB +ultra-f16 slice load-only peak RSS 15.5 MiB +ultra-f16 slice load+vad33s peak RSS 245.5 MiB +redux-keep full load-only peak RSS 372.1 MiB +redux-keep full load+vad33s peak RSS 602.7 MiB +redux-keep slice load-only peak RSS 15.5 MiB +redux-keep slice load+vad33s peak RSS 245.9 MiB +redux-deq-f16 full load-only peak RSS 1382.4 MiB +redux-deq-f16 full load+vad33s peak RSS 1612.4 MiB +redux-deq-f16 slice load-only peak RSS 15.5 MiB +redux-deq-f16 slice load+vad33s peak RSS 246.4 MiB +silero-vad-f16 load-only peak RSS 8.9 MiB +silero-vad-f16 load+vad33s peak RSS 13.3 MiB diff --git a/scripts/vad_bench/results/slice_sha256.txt b/scripts/vad_bench/results/slice_sha256.txt new file mode 100644 index 0000000..7f442a9 --- /dev/null +++ b/scripts/vad_bench/results/slice_sha256.txt @@ -0,0 +1,4 @@ +1b8d46253e2d5205e5058abb7ff12769046001067ddc4892707180cb7a7e22fb redux-deq-f16-vad.gguf +588e1d6e2ee5b6cdfd9ec5ea98dc0993d5bea498d9cc4ec8d6077041eef8a34f redux-keep-vad.gguf +f9df4202f38c528819e663435b21654f44d841f88a0f03d8f48abbb3c00e963e ultra-f16-vad.gguf +8b891a4435e97438104ca07c72530d0c5fe62b986baee48b2dd4e1500c1d4758 ultra-q8_0-vad.gguf diff --git a/scripts/vad_bench/results/slice_speed.json b/scripts/vad_bench/results/slice_speed.json new file mode 100644 index 0000000..99ff2c6 --- /dev/null +++ b/scripts/vad_bench/results/slice_speed.json @@ -0,0 +1,128 @@ +{ + "ultra-q8_0 full|1": [ + 91.6, + 91.6, + 91.0, + 89.4, + 91.4 + ], + "ultra-q8_0 full|8": [ + 266.6, + 251.7, + 250.5, + 185.0, + 203.0 + ], + "ultra-q8_0 slice|1": [ + 91.2, + 91.5, + 91.6, + 89.4, + 91.8 + ], + "ultra-q8_0 slice|8": [ + 257.8, + 248.7, + 222.2, + 227.8, + 221.0 + ], + "ultra-f16 full|1": [ + 93.5, + 92.6, + 91.5, + 92.4, + 94.5 + ], + "ultra-f16 full|8": [ + 253.8, + 255.5, + 233.9, + 255.3, + 254.2 + ], + "ultra-f16 slice|1": [ + 94.3, + 92.5, + 91.9, + 88.5, + 94.6 + ], + "ultra-f16 slice|8": [ + 257.7, + 204.4, + 251.1, + 265.3, + 229.9 + ], + "redux-keep full|1": [ + 94.1, + 92.1, + 91.0, + 93.1, + 94.9 + ], + "redux-keep full|8": [ + 252.8, + 264.0, + 251.3, + 226.8, + 267.1 + ], + "redux-keep slice|1": [ + 94.7, + 90.4, + 92.4, + 91.7, + 95.1 + ], + "redux-keep slice|8": [ + 266.7, + 265.6, + 219.3, + 181.2, + 236.3 + ], + "redux-deq-f16 full|1": [ + 95.1, + 89.8, + 69.8, + 88.4, + 94.0 + ], + "redux-deq-f16 full|8": [ + 259.7, + 254.5, + 264.6, + 265.3, + 260.6 + ], + "redux-deq-f16 slice|1": [ + 94.8, + 88.9, + 84.7, + 89.5, + 93.4 + ], + "redux-deq-f16 slice|8": [ + 268.1, + 261.9, + 263.2, + 263.3, + 260.1 + ], + "silero-vad-f16|1": [ + 736.3, + 724.6, + 661.2, + 687.6, + 702.1 + ], + "silero-vad-f16|8": [ + 473.9, + 483.3, + 459.1, + 485.4, + 462.2 + ] +} \ No newline at end of file diff --git a/scripts/vad_bench/results/timing_rerun_load_log.txt b/scripts/vad_bench/results/timing_rerun_load_log.txt new file mode 100644 index 0000000..504a9a7 --- /dev/null +++ b/scripts/vad_bench/results/timing_rerun_load_log.txt @@ -0,0 +1,21 @@ +22:43:52 start; 22:43:52 up 12 days, 14:48, 1 user, load average: 14.09, 12.73, 10.36 +22:43:52 not quiet: 22:43:52 up 12 days, 14:48, 1 user, load average: 14.09, 12.73, 10.36 +22:53:52 not quiet: 22:53:52 up 12 days, 14:58, 1 user, load average: 38.40, 20.11, 13.44 +23:03:52 not quiet: 23:03:52 up 12 days, 15:08, 1 user, load average: 35.54, 39.78, 28.58 +23:13:52 not quiet: 23:13:52 up 12 days, 15:18, 1 user, load average: 14.53, 19.83, 24.28 +23:23:52 not quiet: 23:23:52 up 12 days, 15:28, 1 user, load average: 34.33, 20.83, 20.40 +23:33:52 not quiet: 23:33:52 up 12 days, 15:38, 1 user, load average: 28.05, 17.41, 17.12 +23:43:52 not quiet: 23:43:52 up 12 days, 15:48, 1 user, load average: 13.25, 12.11, 14.17 +23:53:52 not quiet: 23:53:52 up 12 days, 15:58, 1 user, load average: 82.37, 71.82, 42.46 +00:03:52 not quiet: 00:03:52 up 12 days, 16:08, 1 user, load average: 63.40, 50.75, 45.47 +00:13:52 not quiet: 00:13:52 up 12 days, 16:18, 1 user, load average: 38.12, 53.97, 51.57 +00:23:52 not quiet: 00:23:52 up 12 days, 16:28, 1 user, load average: 32.08, 33.62, 40.68 +00:33:52 not quiet: 00:33:52 up 12 days, 16:38, 1 user, load average: 28.06, 40.43, 41.16 +00:43:52 not quiet: 00:43:52 up 12 days, 16:48, 1 user, load average: 52.52, 43.53, 40.77 +00:53:52 not quiet: 00:53:52 up 12 days, 16:58, 1 user, load average: 7.19, 28.83, 36.45 +01:03:52 not quiet: 01:03:52 up 12 days, 17:08, 1 user, load average: 9.54, 18.41, 27.76 +01:13:52 not quiet: 01:13:52 up 12 days, 17:18, 1 user, load average: 8.63, 17.05, 24.09 +01:23:52 not quiet: 01:23:52 up 12 days, 17:28, 1 user, load average: 6.67, 15.13, 21.99 +01:33:52 not quiet: 01:33:52 up 12 days, 17:38, 1 user, load average: 9.21, 10.77, 16.01 +01:43:52 not quiet: 01:43:52 up 12 days, 17:48, 1 user, load average: 60.44, 35.70, 25.02 +GAVE UP: machine never quiet in 3 h diff --git a/scripts/vad_bench/silero_analyze.py b/scripts/vad_bench/silero_analyze.py new file mode 100755 index 0000000..414da92 --- /dev/null +++ b/scripts/vad_bench/silero_analyze.py @@ -0,0 +1,93 @@ +import json,os,sys,numpy as np +from vad_synth import * +SB=sys.argv[1] # DATA_DIR, as for silero_collect.py; run in the directory that holds probs.npz +P=np.load("probs.npz"); S=json.load(open("segs_own_default.json")) +meta=json.load(open(SB+"/clips.json")) +SYS=["onnx","wcpp_conv623","wcpp_repo620","pk_f32","pk_f16"] +conds=["clean0","clean","white10","pink10"] +def post(p,thr=0.5,min_speech=0.1,min_sil=0.2,pad=0.0,fs=0.032,total=None): + # silero get_speech_timestamps-style hysteresis (neg = thr-0.15), no max-speech split + neg=max(thr-0.15,0.01); segs=[];cur=None;tmp=None + for i,v in enumerate(p): + t=i*fs + if v>=thr and tmp is not None: tmp=None + if v>=thr and cur is None: cur=t; continue + if vmin_speech: segs.append([cur,tmp]) + cur=None;tmp=None + end=len(p)*fs + if cur is not None and end-cur>min_speech: segs.append([cur,end]) + out=[] + for i,(a,b) in enumerate(segs): + out.append([max(0,a-pad),min(total or end,b+pad)]) + return out +def lens(a,b): n=min(len(a),len(b)); return a[:n],b[:n] +# ---- 1. probability parity (vs onnxruntime) ---- +names=[m["name"] for m in meta] +def parity(sysn,ref="onnx",nms=names+["ted_talk"]): + mx=0;sm=0;cnt=0;dec=0;d1=0;big=0 + for n in nms: + a,b=lens(P[f"{n}|{sysn}"],P[f"{n}|{ref}"]); d=np.abs(a-b) + mx=max(mx,d.max());sm+=d.sum();cnt+=len(d);dec+=((a>=.5)!=(b>=.5)).sum();big+=(d>0.05).sum() + return mx,sm/cnt,100*(1-dec/cnt),dec,100*big/cnt +out=[] +out.append("PROBABILITY PARITY vs onnxruntime (v6.2.3 ONNX, official 64-sample context), all 120 clips + 600 s talk, per 32 ms chunk") +out.append("system | max abs diff | mean abs diff | frames > 0.05 off | decision agreement @0.5 | flipped frames") +for s in SYS[1:]: + mx,mn,ag,fl,big=parity(s); out.append(f"{s} | {mx:.4f} | {mn:.5f} | {big:.2f}% | {ag:.3f}% | {fl}") +out.append("pairwise (whisper.cpp converted v6.2.3 vs parakeet f32):") +mx,mn,ag,fl,big=parity("wcpp_conv623","pk_f32");out.append(f"wcpp_conv623 vs pk_f32 | {mx:.4f} | {mn:.5f} | {big:.2f}% | {ag:.3f}% | {fl}") +mx,mn,ag,fl,big=parity("wcpp_conv623","wcpp_repo620");out.append(f"wcpp_conv623 vs wcpp_repo620 | {mx:.4f} | {mn:.5f} | {big:.2f}% | {ag:.3f}% | {fl}") +out.append("per-region parity vs onnx (mean abs diff) : first chunk of each clip / rest") +for s in SYS[1:]: + f=np.mean([abs(P[f"{n}|{s}"][0]-P[f"{n}|onnx"][0]) for n in names]); r=np.mean([np.abs(*[lens(P[f"{n}|{s}"],P[f"{n}|onnx"])[0]-lens(P[f"{n}|{s}"],P[f"{n}|onnx"])[1]])[1:].mean() for n in names]) + out.append(f"{s} first-chunk {f:.5f} rest {r:.5f}") +# ---- 2. segment quality ---- +def agg(getseg,cs): + c=np.zeros(4);bs=[];be=[];miss=0;nb=0 + for m in meta: + if m["cond"] not in cs: continue + L=m["dur"]; pr=getseg(m["name"],L); ref=m["ref"] + c+=counts(mask(pr,L),mask(ref,L)) + e1,m1=bound_err(pr,ref,0);e2,m2=bound_err(pr,ref,1);bs+=e1;be+=e2;miss+=m1+m2;nb+=2*len(ref) + return c,bs,be,miss,nb +sysdefs={} +for s in ["onnx","wcpp_conv623","pk_f32","pk_f16"]: + sysdefs[f"{s} [matched post]"]=(lambda s:lambda n,L:post(P[f"{n}|{s}"],total=L))(s) +sysdefs["wcpp_conv623 [whisper.cpp own default]"]=lambda n,L:S[n]["wcpp_conv623_seg_default"] +sysdefs["wcpp_conv623 [whisper.cpp C++ post, matched params]"]=lambda n,L:S[n]["wcpp_conv623_seg_matched"] +sysdefs["pk_f32 [parakeet own default]"]=lambda n,L:S[n]["pk_f32_seg_default"] +out.append("\nSEGMENT QUALITY, frame level on 10 ms grid vs synthetic reference (P/R/F1 %), 40 clips per condition") +out.append("matched post = thr 0.5 / neg 0.35 / min speech 100 ms / min silence 200 ms / pad 0 (same python code on every probability stream)") +allc=["clean","white10","pink10"] +out.append("system | "+" | ".join(allc)+" | all") +res={} +for k,g in sysdefs.items(): + cells=[] + for cs in [[c] for c in allc]+[allc]: + c,*_=agg(g,cs); p,r,f=prf(c); cells.append(f"{p*100:.1f}/{r*100:.1f}/{f*100:.1f}") + out.append(f"{k} | "+" | ".join(cells)) +out.append("\nBOUNDARY ERROR vs reference, all conditions pooled: start median/p90, end median/p90, miss%") +for k,g in sysdefs.items(): + c,bs,be,miss,nb=agg(g,allc) + out.append(f"{k} | start {np.median(bs):.0f}/{np.percentile(bs,90):.0f} | end {np.median(be):.0f}/{np.percentile(be,90):.0f} | miss {100*miss/nb:.1f}%") +# ---- 4. agreement between implementations ---- +out.append("\nAGREEMENT of speech masks (matched post), all 120 clips pooled: F1 / kappa / IoU") +pairs=[("wcpp_conv623","pk_f32"),("wcpp_conv623","onnx"),("pk_f32","onnx"),("pk_f16","pk_f32"),("wcpp_conv623","wcpp_repo620")] +def agree(a,b,nms,Ls): + c=np.zeros(4) + for n,L in zip(nms,Ls): c+=counts(mask(post(P[f"{n}|{a}"],total=L),L),mask(post(P[f"{n}|{b}"],total=L),L)) + return c +for a,b in pairs: + c=agree(a,b,names,[m["dur"] for m in meta]); p,r,f=prf(c) + out.append(f"{a} vs {b} | F1 {f*100:.2f} | kappa {kappa(c):.4f} | IoU {100*c[0]/(c[0]+c[1]+c[2]):.2f}") +out.append("\nAGREEMENT on the 600 s TED talk (no reference): F1 / kappa, speech fraction") +L=600.0 +for s in ["onnx","wcpp_conv623","pk_f32"]: + m=mask(post(P[f"ted_talk|{s}"],total=L),L); out.append(f"{s} speech fraction {100*m.mean():.1f}%") +for a,b in pairs[:3]: + c=agree(a,b,["ted_talk"],[L]); p,r,f=prf(c); out.append(f"{a} vs {b} | F1 {f*100:.2f} | kappa {kappa(c):.4f}") +out.append("TED talk flipped frames @0.5 vs onnx: "+", ".join(f"{s} {int(((lens(P[f'ted_talk|{s}'],P['ted_talk|onnx'])[0]>=.5)!=(lens(P[f'ted_talk|{s}'],P['ted_talk|onnx'])[1]>=.5)).sum())}/{len(P['ted_talk|onnx'])}" for s in SYS[1:])) +open("results.txt","w").write("\n".join(out)); print("\n".join(out)) diff --git a/scripts/vad_bench/silero_collect.py b/scripts/vad_bench/silero_collect.py new file mode 100755 index 0000000..b335ffe --- /dev/null +++ b/scripts/vad_bench/silero_collect.py @@ -0,0 +1,41 @@ +import json,os,subprocess,sys,numpy as np,soundfile as sf,onnxruntime as ort +# usage: collect.py CORE DATA_DIR MODEL_DIR PARAKEET_CLI WVAD +# DATA_DIR: clips/ clips.json ted_talk.wav (make_clips.py, fetch_data.py) +# MODEL_DIR: silero_vad.onnx silero-vad-f32.gguf silero-vad-f16.gguf ggml-silero-v6.2.3-ggml.bin ggml-silero-v6.2.0.bin +CORE,SB,M,PK,WV=sys.argv[1:6] +TS=["taskset","-c",CORE] +clips=json.load(open(SB+"/clips.json")) +files=[f"{SB}/clips/{c['name']}.wav" for c in clips]+[SB+"/ted_talk.wav"] +names=[c["name"] for c in clips]+["ted_talk"] +so=ort.SessionOptions(); so.intra_op_num_threads=1; so.inter_op_num_threads=1 +sess=ort.InferenceSession(M+"/silero_vad.onnx",so,providers=["CPUExecutionProvider"]) +def onnx_probs(y): + n=(len(y)+511)//512; y=np.pad(y,(0,n*512-len(y))) + st=np.zeros((2,1,128),np.float32); ctx=np.zeros(64,np.float32); out=[] + for i in range(n): + x=np.concatenate([ctx,y[i*512:(i+1)*512]])[None].astype(np.float32) + o,st=sess.run(None,{"input":x,"state":st,"sr":np.array(16000,np.int64)}) + out.append(float(o[0,0])); ctx=x[0,-64:] + return np.array(out) +res={} +def wh(model,tag): + p=subprocess.run(TS+[WV,"probs",model,"1"]+files,capture_output=True,text=True,check=True).stdout.strip().split("\n") + for nm,l in zip(names,p): + j=json.loads(l); res.setdefault(nm,{})[tag]=np.array(j["probs"]); res[nm][tag+"_seg_default"]=j["default"]; res[nm][tag+"_seg_matched"]=j["matched"] +def pk(model,tag): + for nm,f in zip(names,files): + j=json.loads(subprocess.run(TS+[PK,"vad","--model",model,"--input",f,"--probabilities","--threads","1"],capture_output=True,text=True,check=True).stdout) + res.setdefault(nm,{})[tag]=np.array(j["probabilities"]); res[nm][tag+"_seg_default"]=[[s["start"],s["end"]] for s in j["segments"]] +wh(M+"/ggml-silero-v6.2.3-ggml.bin","wcpp_conv623"); print("wcpp conv done",flush=True) +wh(M+"/ggml-silero-v6.2.0.bin","wcpp_repo620"); print("wcpp repo done",flush=True) +pk(M+"/silero-vad-f32.gguf","pk_f32"); print("pk f32",flush=True) +pk(M+"/silero-vad-f16.gguf","pk_f16"); print("pk f16",flush=True) +for nm,f in zip(names,files): + y=sf.read(f,dtype="float32")[0]; res[nm]["onnx"]=onnx_probs(y) +print("onnx done",flush=True) +flat={} +for nm,d in res.items(): + for k,v in d.items(): + if isinstance(v,np.ndarray): flat[f"{nm}|{k}"]=v +np.savez_compressed("probs.npz",**flat) +json.dump({nm:{k:v for k,v in d.items() if not isinstance(v,np.ndarray)} for nm,d in res.items()},open("segs_own_default.json","w")) diff --git a/scripts/vad_bench/slice_bench.py b/scripts/vad_bench/slice_bench.py new file mode 100644 index 0000000..2f86f85 --- /dev/null +++ b/scripts/vad_bench/slice_bench.py @@ -0,0 +1,75 @@ +#!/usr/bin/env python3 +"""Stand-alone head (slice) benchmark: speed, load time and peak memory, full ASR GGUF vs slice GGUF. + +usage: slice_bench.py speed|load|rss +Environment: GGUF_DIR (full GGUFs: ultra-q8_0 ultra-f16 redux-keep redux-deq-f16, plus silero-vad-f16.gguf), +SLICE_DIR (the *-vad.gguf slices of PR 87), WAV300 (300 s clip), CLIP (a short clip for the memory run), +VADBENCH (binary built from vad_slice_bench.cpp inside the PR 87 tree), CLI (parakeet-cli), OUT (output dir). +""" +import subprocess, time, statistics, json, sys, os, re +G=os.environ["GGUF_DIR"]; V=os.environ["SLICE_DIR"]; B=os.environ["VADBENCH"]; WAV=os.environ["WAV300"] +CLIP=os.environ["CLIP"]; CLI=os.environ["CLI"]; OUT=os.environ.get("OUT",".") +S=G +CFG=[("ultra-q8_0 full",G+"/ultra-q8_0.gguf"),("ultra-q8_0 slice",V+"/ultra-q8_0-vad.gguf"), + ("ultra-f16 full",G+"/ultra-f16.gguf"),("ultra-f16 slice",V+"/ultra-f16-vad.gguf"), + ("redux-keep full",G+"/redux-keep.gguf"),("redux-keep slice",V+"/redux-keep-vad.gguf"), + ("redux-deq-f16 full",G+"/redux-deq-f16.gguf"),("redux-deq-f16 slice",V+"/redux-deq-f16-vad.gguf"), + ("silero-vad-f16",G+"/silero-vad-f16.gguf")] +log=open(OUT+"/gate.log","a") +def cpu_idle(): + r={} + for l in open("/proc/stat"): + if l.startswith("cpu") and l[3].isdigit(): + f=l.split(); v=list(map(int,f[1:])); r[int(f[0][3:])]=(v[3]+v[4],sum(v)) + return r +def gate(tag): + a=cpu_idle(); runs=[] + for _ in range(20): + runs.append(int(open("/proc/loadavg").read().split()[3].split("/")[0])-1); time.sleep(0.5) + b=cpu_idle(); load1=float(open("/proc/loadavg").read().split()[0]) + busy={c:1-(b[c][0]-a[c][0])/max(1,b[c][1]-a[c][1]) for c in a} + order=sorted(busy,key=busy.get) + m=statistics.mean(runs); ok=(m<4) or (load1<5) + log.write(f"{time.strftime('%H:%M:%S')} {tag} mean_runnable={m:.1f} load1={load1} ok={ok} idlest={order[:8]}\n"); log.flush() + return ok,order +def runb(args,cpus): + cmd=["flock","/tmp/pk-bench.lock","taskset","-c",",".join(map(str,cpus))]+args + return subprocess.run(cmd,capture_output=True,text=True,check=True).stdout.split() +mode=sys.argv[1] +if mode=="speed": + res={(n,t):[] for n,_ in CFG for t in (1,8)} + for rnd in range(5): + for t in (1,8): + while True: + ok,order=gate(f"speed round{rnd} t{t}") + if ok: break + log.write("not quiet, retry in 30s\n"); log.flush(); time.sleep(30) + cpus=order[:t] + for n,p in CFG: + s,x=runb([B,"speed",p,WAV,str(t)],cpus) + res[(n,t)].append(float(x)) + out={f"{n}|{t}":v for (n,t),v in res.items()} + json.dump(out,open(OUT+"/speed.json","w"),indent=1) + for (n,t),v in res.items(): print(f"{n:22s} {t} thr best {max(v):7.1f}x median {statistics.median(v):7.1f}x all {v}") +elif mode=="load": + out={} + for n,p in CFG: + if len(sys.argv)>2 and not n.startswith(sys.argv[2]): continue + for cold in (False,True): + while True: + ok,order=gate(f"load {n} cold={cold}") + if ok: break + time.sleep(30) + a=[B,"load",p,"10"]+(["cold"] if cold else []) + if not cold: a=[B,"load",p,"11"] # first load warms the page cache and is dropped + v=list(map(float,runb(a,order[:8]))) + if not cold: v=v[1:] + out[f"{n}|{'cold' if cold else 'warm'}"]=v + print(f"{n:22s} {'cold' if cold else 'warm'} median {statistics.median(v):8.1f} ms min {min(v):.1f} max {max(v):.1f}",flush=True) + json.dump(out,open(OUT+"/load%s.json"%("-"+sys.argv[2] if len(sys.argv)>2 else ""),"w"),indent=1) +elif mode=="rss": + for n,p in CFG: + for what,cmd in (("load-only",[B,"load",p,"1"]),("load+vad33s",[CLI,"vad","--model",p,"--input",CLIP,"--threads","8"])): + r=subprocess.run(["/usr/bin/time","-v"]+cmd,capture_output=True,text=True) + m=re.search(r"Maximum resident set size \(kbytes\): (\d+)",r.stderr) + print(f"{n:22s} {what:12s} peak RSS {int(m.group(1))/1024:8.1f} MiB",flush=True) diff --git a/scripts/vad_bench/speed_heads.py b/scripts/vad_bench/speed_heads.py new file mode 100755 index 0000000..0caffb6 --- /dev/null +++ b/scripts/vad_bench/speed_heads.py @@ -0,0 +1,18 @@ +#!/usr/bin/env python3 +"""usage: PARAKEET_LIB=... taskset -c CPUS speed_heads.py TALK.wav GGUF_DIR +VAD speed on the first 300 s: Silero ONNX on 1 thread vs the heads (8 threads), 4 reps.""" +import sys,time,subprocess,numpy as np,soundfile as sf,torch +from heads_lib import silero,parakeet +from pk import PK +y=sf.read(sys.argv[1],dtype="float32")[0][:16000*300] # 300 s +D=len(y)/16000 +pks={"ultra_q8":PK(sys.argv[2]+"/ultra-q8_0.gguf"),"ultra_f16":PK(sys.argv[2]+"/ultra-f16.gguf"),"redux_packed":PK(sys.argv[2]+"/redux-keep.gguf"),"redux_deq_f16":PK(sys.argv[2]+"/redux-deq-f16.gguf")} +fns={"silero_onnx_1thr":lambda:silero(y,True)} +for n,p in pks.items(): fns[n+"_8thr"]=(lambda p=p:parakeet(p,y)) +T={k:[] for k in fns}; loads=[] +for rep in range(4): + loads.append(subprocess.getoutput("cut -d' ' -f1 /proc/loadavg")) + for k,f in fns.items(): + t0=time.perf_counter();f();T[k].append(time.perf_counter()-t0) +print("load1 at each rep start:",loads) +for k,v in T.items(): print("%-18s best %.2fs median %.2fs -> %.0fx / %.0fx real time"%(k,min(v),np.median(v),D/min(v),D/np.median(v))) diff --git a/scripts/vad_bench/speed_silero.py b/scripts/vad_bench/speed_silero.py new file mode 100755 index 0000000..98c83f3 --- /dev/null +++ b/scripts/vad_bench/speed_silero.py @@ -0,0 +1,71 @@ +#!/usr/bin/env python3 +"""Timing of Silero VAD on a 300 s clip, one pinned core, serialized with a lock. + +usage: speed_silero.py CORE ROUNDS WAV300 MODEL_DIR PARAKEET_CLI WVAD WHISPER_BIN_DIR OUT.json [--lock FILE] + +Jobs (round-robin order, one run each per round): + wcpp_inproc_detect_only wvad time: whisper_vad_detect_speech only, model already loaded + wcpp_vad-speech-segments_cli whisper-vad-speech-segments (model load + detect + print) + parakeet_cli_f32 / _f16 parakeet-cli vad (model load + detect + print) + onnxruntime_python_1thr_detect_only onnx_time.py (Python loop, one 512-sample call per chunk) + whisper-cli_tiny.en_novad / _vad whisper-cli end to end with tiny.en, 1 thread (needs ggml-tiny.en.bin) +Before each run the script waits until the mean number of other runnable tasks over 10 s is below +4 or the 1-minute load average is below 5 (it gives up waiting after 30 tries and runs anyway, and +records quiet=false). It reports wall time, user time, max RSS (from /usr/bin/time -v), the mean +runnable count and the load average at the start. MODEL_DIR: see silero_collect.py. +""" +import fcntl, json, os, re, subprocess, sys, tempfile, time +import numpy as np + +CORE, ROUNDS, WAV, M, PK, WV, WB, OUT = sys.argv[1], int(sys.argv[2]), *sys.argv[3:9] +LOCK = sys.argv[sys.argv.index("--lock") + 1] if "--lock" in sys.argv else "/tmp/pk-bench.lock" +WS, WC = WB + "/whisper-vad-speech-segments", WB + "/whisper-cli" +VM = M + "/ggml-silero-v6.2.3-ggml.bin"; TM = M + "/ggml-tiny.en.bin" +HERE = os.path.dirname(os.path.abspath(__file__)) +JOBS = { + "wcpp_inproc_detect_only": [WV, "time", VM, "1", "1", WAV], + "wcpp_vad-speech-segments_cli": [WS, "-vm", VM, "-f", WAV, "-t", "1", "-np"], + "parakeet_cli_f32": [PK, "vad", "--model", M + "/silero-vad-f32.gguf", "--input", WAV, "--threads", "1"], + "parakeet_cli_f16": [PK, "vad", "--model", M + "/silero-vad-f16.gguf", "--input", WAV, "--threads", "1"], + "onnxruntime_python_1thr_detect_only": ["python3", HERE + "/onnx_time.py", M + "/silero_vad.onnx", WAV], +} +if os.path.exists(TM): + JOBS["whisper-cli_tiny.en_novad"] = [WC, "-m", TM, "-f", WAV, "-t", "1", "-np", "-nt"] + JOBS["whisper-cli_tiny.en_vad"] = [WC, "-m", TM, "-f", WAV, "-t", "1", "-np", "-nt", "--vad", "-vm", VM] +INPROC = ("wcpp_inproc", "onnxruntime") + +def runnable(): + return int(open("/proc/loadavg").read().split()[3].split("/")[0]) - 1 # minus this process + +def gate(): + s = [] + for _ in range(10): + s.append(runnable()); time.sleep(1) + return float(np.mean(s)), float(open("/proc/loadavg").read().split()[0]) + +lockf = open(LOCK, "a") +tf = tempfile.NamedTemporaryFile(suffix=".time", delete=False).name + +def timed(name): + cmd = ["taskset", "-c", CORE, "/usr/bin/time", "-v", "-o", tf] + JOBS[name] + for attempt in range(30): + fcntl.flock(lockf, fcntl.LOCK_EX) + try: + mr, l1 = gate(); quiet = (mr < 4 or l1 < 5) + if quiet or attempt == 29: + t0 = time.perf_counter(); p = subprocess.run(cmd, capture_output=True, text=True); wall = time.perf_counter() - t0 + tt = open(tf).read() + rss = int(re.search(r"Maximum resident set size \(kbytes\): (\d+)", tt).group(1)) + user = float(re.search(r"User time \(seconds\): ([\d.]+)", tt).group(1)) + inproc = float(p.stdout.strip().split()[0]) if name.startswith(INPROC) else None + return dict(job=name, wall=wall, inproc=inproc, rss_kb=rss, user=user, mean_runnable=mr, load1=l1, quiet=quiet, attempts=attempt + 1) + finally: + fcntl.flock(lockf, fcntl.LOCK_UN) + time.sleep(20) + +names = list(JOBS); rows = [] +for r in range(ROUNDS): + for n in names[r % len(names):] + names[:r % len(names)]: + x = timed(n); x["round"] = r; rows.append(x); print(json.dumps(x), flush=True) +os.unlink(tf) +json.dump(rows, open(OUT, "w"), indent=1) diff --git a/scripts/vad_bench/ted_reference.py b/scripts/vad_bench/ted_reference.py new file mode 100755 index 0000000..a0ecbad --- /dev/null +++ b/scripts/vad_bench/ted_reference.py @@ -0,0 +1,19 @@ +#!/usr/bin/env python3 +"""usage: ted_reference.py DATA_DIR TDT_0.6B_V3_F16.gguf +Word timestamps of each talk from the ASR model (30 s clips, batches of 8); compare_ted.py merges +words closer than 0.4 s into speech spans. The reference is therefore coarse and ASR-derived.""" +import json,glob,sys,soundfile as sf,numpy as np +from pk import PK +pk=PK(sys.argv[2]) +for f in sorted(glob.glob(sys.argv[1]+"/ted_*.wav")): + y,_=sf.read(f,dtype="float32") + if len(y)<16000*60: continue + W=[];C=30*16000 + clips=[y[i:i+C] for i in range(0,len(y),C)] + for b in range(0,len(clips),8): + res=pk.words_batch(clips[b:b+8]) + for k,r in enumerate(res): + off=(b+k)*30.0 + for w in r["words"]: W.append([w["start"]+off,w["end"]+off]) + json.dump(W,open(f.replace(".wav","_words.json"),"w")) + print(f,len(y)/16000,len(W),flush=True) diff --git a/scripts/vad_bench/vad_slice_bench.cpp b/scripts/vad_bench/vad_slice_bench.cpp new file mode 100644 index 0000000..ec261c6 --- /dev/null +++ b/scripts/vad_bench/vad_slice_bench.cpp @@ -0,0 +1,44 @@ +// load N [cold] : N timed loads (ms each). speed : one timed VAD run after a warm-up. +#include +#include +#include +#include +#include +#include +#include +#include "model.hpp" +#include "silero_vad.hpp" +#include "vad_json.hpp" +#include "audio_io.hpp" +#include "ggml_graph.hpp" +using clk = std::chrono::steady_clock; +struct H { std::unique_ptr m; std::unique_ptr s; }; +static H load(const std::string& p) { + H h; std::string err; + if (pk::gguf_is_silero(p)) h.s = pk::SileroVad::load(p, &err); + else h.m = pk::gguf_is_vad_only(p) ? pk::Model::load_vad_only(p) : pk::Model::load(p); + if (!h.m && !h.s) { std::fprintf(stderr, "load failed\n"); std::exit(1); } + return h; +} +static std::string run(const H& h, const std::vector& pcm) { + pk::VadRequest r; r.kind = h.s ? pk::VadKind::kSilero : pk::VadKind::kHead; r.opts = pk::default_segmenter_opts(r.kind); + return h.s ? pk::silero_vad_to_json(*h.s, pcm, 16000, r) : pk::vad_to_json(*h.m, pcm, r); +} +int main(int argc, char** argv) { + std::string mode = argv[1], p = argv[2]; + if (mode == "load") { + int n = atoi(argv[3]); bool cold = argc > 4; + for (int i = 0; i < n; ++i) { + if (cold) { int fd = open(p.c_str(), O_RDONLY); posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED); close(fd); } + auto t0 = clk::now(); { H h = load(p); } + std::printf("%.1f\n", std::chrono::duration(clk::now() - t0).count()); + } + } else { + pk::Audio a; pk::load_audio_16k_mono(argv[3], a); + pk::set_num_threads(atoi(argv[4])); + H h = load(p); run(h, a.samples); + auto t0 = clk::now(); std::string o = run(h, a.samples); + double s = std::chrono::duration(clk::now() - t0).count(); + std::printf("%.4f %.1f\n", s, a.samples.size() / 16000.0 / s); + } +} diff --git a/scripts/vad_bench/vad_synth.py b/scripts/vad_bench/vad_synth.py new file mode 100755 index 0000000..b27114c --- /dev/null +++ b/scripts/vad_bench/vad_synth.py @@ -0,0 +1,46 @@ +import numpy as np +GR=0.01 # mask grid 10 ms +def mask(spans,n_sec): + m=np.zeros(int(round(n_sec/GR)),bool) + for s,e in spans: + m[int(round(s/GR)):int(round(e/GR))]=True + return m +def trim(u): + fr=len(u)//160; r=np.sqrt((u[:fr*160].reshape(fr,160)**2).mean(1)) + idx=np.where(r>r.max()*0.01)[0] + return u[idx[0]*160:(idx[-1]+1)*160] +def pink(n,rng): + X=np.fft.rfft(rng.standard_normal(n)); f=np.arange(len(X)); f[0]=1 + y=np.fft.irfft(X/np.sqrt(f),n); return y/y.std() +def build(utts,rng): + g=lambda a,b: rng.uniform(a,b) + parts=[];spans=[];t=g(0.5,1.5) + parts.append(rng.standard_normal(int(t*16000))*1e-3) + for i,u in enumerate(utts): + u=trim(u); spans.append((t,t+len(u)/16000)); parts.append(u); t+=len(u)/16000 + gap=g(0.3,2.0) if i1.0: miss+=1 + else: out.append(d*1000) + return out,miss + diff --git a/scripts/vad_bench/wvad.cpp b/scripts/vad_bench/wvad.cpp new file mode 100644 index 0000000..b4c5370 --- /dev/null +++ b/scripts/vad_bench/wvad.cpp @@ -0,0 +1,65 @@ +// Harness over whisper.cpp's public whisper_vad_* API. Not part of whisper.cpp. +// wvad probs ... -> one JSON line per wav (probs + segments) +// wvad time -> detect_speech seconds per rep +#include "whisper.h" +#include +#include +#include +#include +#include +#include +static bool read_wav16(const char* p, std::vector& out) { + FILE* f = fopen(p, "rb"); if (!f) return false; + char id[4]; uint32_t sz; fread(id, 1, 4, f); fread(&sz, 4, 1, f); fread(id, 1, 4, f); + while (fread(id, 1, 4, f) == 4 && fread(&sz, 4, 1, f) == 1) { + if (!memcmp(id, "data", 4)) { + std::vector d(sz / 2); fread(d.data(), 2, d.size(), f); fclose(f); + out.resize(d.size()); for (size_t i = 0; i < d.size(); i++) out[i] = d[i] / 32768.0f; return true; + } + fseek(f, sz, SEEK_CUR); + } + fclose(f); return false; +} +static void segs(whisper_vad_context* v, whisper_vad_params p, const char* key) { + whisper_vad_segments* s = whisper_vad_segments_from_probs(v, p); + printf("\"%s\":[", key); + int n = s ? whisper_vad_segments_n_segments(s) : 0; + for (int i = 0; i < n; i++) + printf("%s[%.2f,%.2f]", i ? "," : "", whisper_vad_segments_get_segment_t0(s, i) / 100.0, whisper_vad_segments_get_segment_t1(s, i) / 100.0); + printf("]"); if (s) whisper_vad_free_segments(s); +} +static void quiet(enum ggml_log_level, const char*, void*) {} +int main(int argc, char** argv) { + whisper_log_set(quiet, nullptr); + if (argc < 5) return 1; + std::string mode = argv[1]; + whisper_vad_context_params cp = whisper_vad_default_context_params(); + cp.n_threads = atoi(argv[3]); cp.use_gpu = false; + whisper_vad_context* v = whisper_vad_init_from_file_with_params(argv[2], cp); + if (!v) { fprintf(stderr, "load failed\n"); return 2; } + std::vector y; + if (mode == "time") { + int reps = atoi(argv[4]); + if (!read_wav16(argv[5], y)) return 3; + for (int r = 0; r < reps; r++) { + auto t0 = std::chrono::steady_clock::now(); + whisper_vad_detect_speech(v, y.data(), (int)y.size()); + double dt = std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); + printf("%.4f\n", dt); + } + return 0; + } + for (int a = 4; a < argc; a++) { + if (!read_wav16(argv[a], y)) return 3; + if (!whisper_vad_detect_speech(v, y.data(), (int)y.size())) return 4; + int n = whisper_vad_n_probs(v); float* pr = whisper_vad_probs(v); + printf("{\"name\":\"%s\",\"n\":%d,\"probs\":[", argv[a], n); + for (int i = 0; i < n; i++) printf("%s%.6f", i ? "," : "", pr[i]); + printf("],"); + segs(v, whisper_vad_default_params(), "default"); printf(","); + whisper_vad_params m = whisper_vad_default_params(); + m.threshold = 0.5f; m.min_speech_duration_ms = 100; m.min_silence_duration_ms = 200; m.speech_pad_ms = 0; + segs(v, m, "matched"); printf("}\n"); + } + whisper_vad_free(v); +}