diff --git a/README.md b/README.md index 5621ca0..7fb3c20 100644 --- a/README.md +++ b/README.md @@ -85,6 +85,14 @@ ternary ones. functions, as a stream (`parakeet_capi_vad_stream_*`), and as the cutter for any ASR model: `parakeet-cli transcribe --vad --vad-model silero.gguf`. See [`docs/vad.md`](docs/vad.md). - VAD-only slices of Ultra and Redux (6 to 10 MB, `redux-vad.gguf` and `ultra-vad-q8_0.gguf` in the same repo) hold just the head and its front end. They run `vad` and the `parakeet_capi_vad_*` calls and cannot transcribe. See [`docs/vad.md`](docs/vad.md). +- The head alone gives false alarms on audio without speech: on speech-free noise it calls about + 99 percent of the frames speech (Ultra 99.4, Redux 97.8), and over a 30 s noise stretch inside a + file with speech the false-alarm frame rate was 17.7 percent for Ultra and 55 percent for Redux, + against 0 percent for Silero (synthetic LibriSpeech with added noise). Prefer Silero as the + always-on gate or when the audio can have long non-speech stretches; use the head on audio known + to be mostly speech, or where its higher recall matters. An offline experiment that lets Silero + decide and the head move the edges (not implemented here) is in + [`docs/vad-benchmarks.md`](docs/vad-benchmarks.md#fusing-silero-and-the-head-offline-experiment). - Accuracy, speed and size of both detectors, with the method and the scripts to repeat them: [`docs/vad-benchmarks.md`](docs/vad-benchmarks.md). --- diff --git a/docs/vad-benchmarks.md b/docs/vad-benchmarks.md index ac02032..fd91aa5 100644 --- a/docs/vad-benchmarks.md +++ b/docs/vad-benchmarks.md @@ -548,6 +548,383 @@ one command each when the machine is quiet. The accuracy numbers do not depend on load and were not re-run. +## The head on noise-only audio + +The Parakeet VAD head gives false alarms on audio that has no speech. Silero does not. +This was first measured on synthetic data: LibriSpeech test-clean utterances with added +white or pink noise (see [the fusion experiment](#fusing-silero-and-the-head-offline-experiment) +for the corpus). A follow-up root-cause study (below) then looked at why, at other noise +types and levels, and at what it costs in `transcribe --vad`. The numbers in the first +two tables are frame rates at threshold 0.5 with the default segmenter settings (bridge +0.1 s, minimum speech 0.1 s, minimum pause 0.2 s, no padding). + +| Test | Silero | Ultra head (Q8_0) | Redux head (packed) | +| --- | ---: | ---: | ---: | +| Speech-free 30 s clips (96 clips: white and pink noise at 20, 10, 5 and 0 dB), share of frames called speech | 0.00% | 99.44% | 97.83% | +| A 30 s noise stretch inside a file that also has speech (96 files), share of frames in the stretch called speech | 0.0% | 17.7% | 55.2% | + +For the stretch inside a speech file, the rate depends on the noise level: + +| Noise level (SNR) | Ultra head | Redux head | Silero | +| --- | ---: | ---: | ---: | +| 20 dB | 0.0% | 35.2% | 0.0% | +| 10 dB | 0.6% | 65.3% | 0.0% | +| 5 dB | 14.3% | 60.8% | 0.0% | +| 0 dB | 55.8% | 59.4% | 0.0% | + +Each cell pools 12 white noise and 12 pink noise files. The numbers are in +`scripts/vad_bench/fusion/results.md` (noise-only clips) and `gap_results.md` (the +stretch inside a file). + +### Root-cause study + +Verdict: this is not a bug in parakeet.cpp. It is a property of the model. Scripts and +result files: [`scripts/vad_bench/noise_dive/`](../scripts/vad_bench/noise_dive/README.md). + +**Our implementation is faithful.** An independent reference was built from Hugging Face +transformers (feature extractor and subsampler) and the head structure documented in +[ternary.md](ternary.md). On 8 signals it matches `parakeet-cli vad --probabilities` to a +maximum probability difference of 1.1e-4 to 2.6e-4 on F16 files (Redux and Ultra F16), +and up to 1.1e-2 for the Ultra Q8_0 file (quantisation), and the speech +fractions agree (`res_A.md`). On the 12 stretch files checked, the CLI and the reference +agree on the false-alarm rate to within 0.5 points (`res_cli_check.txt`). The one real +difference is digital silence (see Proposed behaviour). + +#### Why + +The head judges each 80 ms frame by its level and texture relative to the average of the +file it is in. Steady loud noise, or a file that is only noise, therefore sits at a logit +of about +1.5 to +2 (probability 0.8 to 0.9). Speech sits at +5 to +12 and pauses at -3 +to -8: + +| Logit (median, 5th to 95th percentile) | Redux | Ultra | +| --- | ---: | ---: | +| Speech in clean clips | +8.0 (-3.1 to +12.7) | +7.0 (-2.7 to +13.2) | +| Pauses in clean clips (digital -60 dBFS dither) | -8.1 (-11.6 to -5.5) | -3.2 (-5.0 to -1.8) | +| Noise-only file at -23 dBFS | +1.5 (+0.2 to +3.0) | +1.7 (+0.5 to +3.1) | +| Noise-only file at -50 dBFS | +0.5 (-0.6 to +1.8) | +1.3 (+0.1 to +2.6) | + +![Logit histograms](../scripts/vad_bench/noise_dive/fig_logit_hist.png) + +What the study found about the cause: + +- The per-file normalisation of the features is one input, not the whole cause. With a + fixed normalisation (mean and standard deviation taken from the speech clips), noise-only + files at -60 and -50 dBFS stop firing (Redux 0.0 percent, Ultra 0.0 to 49 percent + depending on the statistics), but at -30 and -23 dBFS they still do (Redux 78 and 97 + percent with one set of statistics). A noise stretch inside a speech file does not go away + either (`res_B3.txt`). +- Level alone does not explain the logit on noise. A held-out regression of the logit on + six frame features (level, absolute log level, flux, flatness, 2 to 8 Hz modulation, + tilt) gives R2 of 0.01 to 0.03 on white, pink and speech-shaped noise (about 0.2 on + brown noise) and 0.74 to 0.76 on speech (`res_B4.txt`). Inside a speech file the logit + follows the mean level of the stretch relative to the file (correlation 0.80 for Redux + and 0.84 for Ultra with the median logit, `res_B8.txt`) and its temporal variation + (`res_B10.txt`). +- The share of noise in the file is not monotonic. For white noise at -15 dB re the talk, + Ultra calls 100 percent of a 5 s stretch speech, 23 percent of a 60 s stretch and 3 + percent of a 90 s stretch (`res_B7.md`). +- Amplitude invariance does not hold. Above about -40 dBFS RMS the output is flat. + Below it the output falls off (white noise alone in a file: Redux 94 percent speech + frames at -40 dBFS, 22 at -60, 0 at -80; Ultra 99, 85, 7; `res_B1.txt`), because of the + 2^-24 guard inside the log of the mel features. That attribution is from the study + notes; no result file isolates the guard. +- Moondream's cards say the head exists so that recordings can be cut at pauses into + segments of at most 30 s. What it was trained on is not stated. Nothing in the cards + claims robustness to noise or music. Moondream's own runtime was not available, so the + head wiring is inferred from the documentation and is consistent with the behaviour + (a residual variant of the context block breaks Ultra: clean speech F1 81.0 instead of + 95.0, `res_A2_variants.txt`). + +#### What the head does with different noise + +Each signal is the only content of a 30 s file (375 frames), share of frames called +speech at threshold 0.5 (`res_B2.md`): + +| Signal | Redux | Ultra | +| --- | ---: | ---: | +| Digital silence | 98.7% | 99.7% | +| Constant tone (1 kHz, -20 dBFS) | 0.0% | 98.9% | +| Sine sweep 50 to 7500 Hz | 5.9% | 45.1% | +| Clicks, 3 per second | 3.5% | 46.9% | +| Clicks, 10 per second | 33.1% | 94.4% | +| Hum 50 Hz and harmonics, -25 dBFS | 0.0% | 0.0% | +| Music-like chords, -20 dBFS | 23.5% | 54.9% | +| Brown noise, -25 dBFS | 28.3% | 19.2% | +| White noise, -25 dBFS | 98.7% | 99.2% | +| Pink noise, -25 dBFS | 97.3% | 98.4% | +| Speech (LibriSpeech, for comparison) | 91.5% | 88.5% | + +Redux and Ultra differ by stimulus, and neither is simply worse. Redux reacts to the +texture of flat-spectrum noise (white, pink, speech-shaped) and reacts much less to +tones, hum, sparse clicks and music. Ultra reacts to level, and also to clicks, sweeps +and music, but it stops firing on white and pink noise sooner once the noise is below +the speech. A pure tone fires Ultra only when it is alone in a file (98.9 percent above); +inside a speech file it gave 0 percent (next table). + +#### How far below the speech the noise must be + +In a speech file, with a 30 s stretch of a given noise type, share of stretch frames +called speech by the level of the stretch below the speech (6 clips per cell, +`res_B6.txt`): + +| Stretch type | Redux 10 dB | Redux 20 dB | Redux 30 dB | Ultra 10 dB | Ultra 20 dB | Ultra 30 dB | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| White | 79 | 53 | 0 | 0 | 0 | 0 | +| Pink | 63 | 33 | 0 | 0 | 0 | 0 | +| Speech-shaped | 48 | 43 | 1 | 46 | 0 | 0 | +| Clicks, 10 per second | 16 | 9 | 0 | 64 | 22 | 1 | +| Clicks, 3 per second | 1 | 1 | 0 | 26 | 9 | 0 | +| Music-like | 16 | 10 | 1 | 44 | 8 | 0 | +| Brown noise, 50 Hz hum, 1 kHz sine | 0 | 0 | 0 | 0 | 0 | 0 | + +![Stretch false alarms against level](../scripts/vad_bench/noise_dive/fig_types_vs_level.png) + +- Brown noise, hum and a pure tone never trigger the head inside a speech file, for + either model. +- Redux needs the white or pink stretch to be 30 dB below the speech before it stops + (it is still 33 to 53 percent at 20 dB). Ultra stops at 10 dB for white and pink, 20 dB + for speech-shaped noise, and 30 dB for clicks and music. +- Alone in a file, white noise stops being called speech below about -55 dBFS for Redux + and about -70 dBFS for Ultra (`res_B1.txt`; the grid has 10 dB steps: Redux 22 percent + at -60 and 8 at -70, Ultra 42 at -70 and 7 at -80). + +This corrects the earlier statement that false alarms grow with the noise level. That is +true for Ultra (stretch in a speech file, white and pink pooled: 0 percent at 20 dB SNR, 14 +at 5 dB, 56 at 0 dB). It is not true for Redux, which is already at 35 percent at 20 dB +and stays at 59 to 65 percent from 10 dB to 0 dB. And for both models the false-alarm +rate is not monotonic in the share of the file that is noise (`res_B7.md`). + +#### The user-visible cost in `transcribe --vad` + +A 60 s block of inserted audio (white, pink, clicks, music; at -35, -20 and -5 dB +relative to the RMS of the talk) was put at 111.7 to 171.7 s of a 300 s excerpt of a real +talk. Words that start inside the block are hallucinations, because the block has no +speech (`res_D.md`, 24 runs per model): + +| Case | Hallucinated words in the block | +| --- | ---: | +| Without VAD, all 24 runs (12 per model) | 0 | +| `--vad`, Redux | 0 in 11 files, 2 for white noise at -20 dB | +| `--vad`, Ultra | 0 in 9 files, 15, 12 and 3 for music at -35, -20 and -5 dB | +| `--vad`, Silero (Ultra decoder, music at -35 dB, the only Silero file transcribed) | 20 | + +Example text: "I don't know, I don't know, I don't know" for Ultra on music at -35 dB. The +Silero case kept an 18 s segment of music because its 30 s segment ended 18 s into the +block (`res_D_silero.txt`). + +Per-frame head calls inside the inserted block (`res_D_probs.md`): Redux at most 12 +percent in any file; Ultra 68 percent for white at -5 dB and 38 percent for music and +clicks at -5 dB. At -35 dB the head called 0 percent of the block speech in 5 of the 8 files. +Even so, the segments the head produced covered 24 to 60 s of the 60 s block (Redux 24 +to 60 s, Ultra 54 to 60 s) and Silero about 18 s. + +The cause is the segmenter, not the head. With a 60 s block the head's segments still +covered 54 to 60 s of it in nearly every file, even where the head called 0 percent of the +block speech (Ultra white and music at -35 dB). A hard 30 s cut can land just before +speech resumes, so a piece with a little speech at its edge is kept and decoded. On the +two Silero files checked (white at -20 dB, music at -35 dB) Silero dropped about 42 s of the +block. This segmenter behaviour deserves its own look. It is not fixed. + +So the user-visible cost is small: wasted decoding of segments that are noise, and rarely +a hallucinated phrase. With `--vad` the talk words outside the block differed from the +run without VAD by at most 6 words in about 814 (`res_D.md`). The head does well on real speech: on a 1168 s talk it called speech 97.0 +percent (Redux) and 96.3 percent (Ultra) of the time, where Silero called 93.4 percent, +and the head and Silero agree with an F1 of 98.0 to 98.3 (`res_E2_talk.txt`). On +synthetic clips with pauses its frame F1 is 95.4 (Redux) and 95.0 (Ultra), and 94.7 and +93.6 with added white and pink noise (`res_E.md`). + +#### Mitigations we tested + +Offline, on a Python reimplementation of the segmenter on a 10 ms grid, with the speech +clips and noise of the fusion corpus (`res_E.md`). F1 is the frame F1 on speech; the +noisy set is white and pink noise at 20 to 0 dB. False alarms are the share of frames +called speech in noise-only files and in a 30 s stretch embedded in speech. Lower is +better for false alarms. + +| Rule | Redux F1 clean / noisy | Redux false alarms, noise-only / stretch | Ultra F1 clean / noisy | Ultra false alarms, noise-only / stretch | +| --- | --- | --- | --- | --- | +| Head, threshold 0.5 (default) | 95.4 / 94.7 | 97.8 / 55.2 | 95.0 / 93.6 | 99.4 / 17.3 | +| Head, threshold 0.7 | 94.8 / 93.9 | 76.5 / 10.1 | 94.2 / 91.8 | 94.2 / 3.6 | +| Head, threshold 0.9 | 93.9 / 92.0 | 16.3 / 0.1 | 92.4 / 87.5 | 25.8 / 0.0 | +| Head, threshold 0.97 | 92.5 / 89.4 | 1.0 / 0.0 | 90.0 / 80.6 | 0.5 / 0.0 | +| Head 0.5, minimum speech 1.0 s | 94.6 / 94.1 | 96.6 / 29.0 | 93.8 / 91.9 | 99.4 / 6.9 | +| Head 0.5, energy gate (frame at least 25 dB below the file P95) | 92.1 / 94.3 | 97.8 / 53.2 | 92.0 / 93.3 | 99.4 / 17.3 | +| Head 0.5, keep a run only if its median logit is at least 2.5 (p about 0.92) | 95.4 / 94.7 | 0.1 / 0.3 | 94.5 / 90.3 | 0.0 / 0.0 | +| Silero 0.5 AND head 0.5 | 93.4 / 91.8 | 0.0 / 0.0 | 93.3 / 91.2 | 0.0 / 0.0 | +| Two-stage (Silero decides, head moves the edges) | 95.4 / 94.8 | 0.0 / 0.0 | 95.1 / 94.2 | 0.0 / 0.0 | +| Silero 0.5 alone, for reference | 93.9 / 92.4 | 0.0 / 0.0 | 93.9 / 92.4 | 0.0 / 0.0 | + +In plain language: + +- Raising the threshold trades recall for fewer false alarms and still leaves noise-only + files firing until about 0.97, where speech F1 has fallen by 3 to 5 points (clean) and + up to 13 points (Ultra, noisy). At 0.9 most false alarms in a speech file go away, at a + cost of 1.5 to 2.6 F1 points on clean speech. +- The energy gate and a longer minimum speech do not help: the gate removes speech + (clean F1 86.1 to 94.6, depending on the gate) and leaves noise-only files at 97.8 and 99.4 percent. A minimum speech of + 1.0 s helps only for the embedded stretch (Ultra 17.3 to 6.9) and costs about 0.8 to 1.2 + clean F1 points. +- The median-logit run gate is the best cheap option. It drops almost all false alarms + and costs nothing for Redux on clean speech (95.4) and 0.5 F1 points for Ultra (3.3 on the + noisy set). With the run threshold at 3.5 (not in the table) Redux clean F1 is 94.9 and + Ultra 93.6. It is not validated on WER and is not + implemented. On the 1168 s talk it removed 0.2 to 0.3 percent of the head's speech + frames at 2.5 (`res_E2_talk.txt`). In the inserted block test at most 18 percent of the + block frames remained, and 0 in 21 of 24 files (`res_D_mit.md`). +- The two-stage rule needs Silero. It removes the false alarms and keeps the F1 of the head. + See [the fusion experiment](#fusing-silero-and-the-head-offline-experiment). + +#### Proposed behaviour + +The defaults stay as they are. One candidate fix, not in this PR: a guard for digital +silence or a constant tone. Our CLI calls 98.7 percent (Redux) and 99.7 percent (Ultra) +of the frames of a silent file speech, because it accumulates the normalisation in +double and gets exactly zero features. The reference with the same double accumulation +gives a speech probability of 0.76 to 0.86 per frame on the same file (`res_B2.md` +column "mean p"). The Hugging Face float32 normalisation leaves rounding noise instead and +gives a probability of about 0; that float32 run is not among the committed result +files. + +#### Limits of this study + +- Moondream's runtime was not available. The head wiring is inferred from the + documentation and is consistent with the behaviour, but not checked against it. +- The noise is synthetic: LibriSpeech with generated white, pink, brown, speech-shaped, + click, hum, tone and music-like signals, plus one real talk. +- The mitigation numbers come from a Python reimplementation of the segmenter on a 10 ms + grid, not from the C++ code. +- The hallucination counts come from one talk and one insertion point. Only one Silero + file was transcribed. +- The black-box check of the head against the released runtime was skipped (it needs + CUDA or Metal). + +## Fusing Silero and the head (offline experiment) + +The head has the higher recall and Silero has the cleaner decisions. This experiment +asked whether combining them beats each alone. **Fusion is not implemented in +parakeet.cpp.** It is an offline experiment: the rules run in a Python port of the +segmenter (`scripts/vad_bench/fusion/`), on probabilities saved from Silero and from the +heads. No C++ code, API or option changed. + +### Setup + +- Corpus: rebuilt from LibriSpeech test-clean. 342 speech clips (38 sets of 5 + utterances, each set in 9 conditions: clean, and white and pink noise at 20, 10, 5 and + 0 dB SNR) and 96 noise-only 30 s clips (12 per noise type and level). +- Detectors: Silero ONNX 6.2.3 (run with onnxruntime), the Ultra Q8_0 head and the Redux + packed head, through `parakeet-cli vad --probabilities`. +- All probabilities go to a 10 ms grid by sample-and-hold. All rules use the same + post-processing: bridge 0.1 s, minimum speech 0.1 s, minimum pause 0.2 s, no padding. +- Tuned numbers use speaker-disjoint 4-fold cross-validation: the parameters are chosen on + three folds and scored on the fourth. All intervals are 95 percent bootstrap intervals + over clips. +- Score: frame level precision (P), recall (R) and F1 against the reference, pooled over + the 342 speech clips. The reference is the span of each utterance in the synthetic + clip, with an energy trim. + +### The two-stage rule + +Silero decides what is speech. The head only changes the edges: it moves the start of a +Silero region earlier by up to 160 to 320 ms and its end later by up to 160 ms, but only +where the head also says speech, and it fills a gap between two regions if the gap is +shorter than 300 to 600 ms and the head calls at least half of it speech. The default +setting is 160 ms before, 160 ms after, 300 ms fill; the best setting found on all the +clips was 320 ms before, 160 ms after, 600 ms fill. Where Silero finds no speech, the head can +never add any. This is why the rule has no false alarms on noise. + +### Results at the defaults (threshold 0.5) + +Pooled over 342 clips, percent. + +| System | P | R | F1 | +| --- | ---: | ---: | ---: | +| Silero | 99.6 | 85.6 | 92.1 | +| Ultra head | 98.6 | 89.4 | 93.8 | +| Redux head | 97.6 | 92.0 | 94.7 | +| OR of both (Redux) | 97.6 | 92.1 | 94.8 | +| AND of both (Redux) | 99.6 | 85.5 | 92.0 | +| Two-stage (Ultra) | 99.0 | 90.0 | 94.3 | +| Two-stage (Redux) | 98.9 | 91.2 | 94.9 | + +### Results after tuning (cross-validated) + +| System | P | R | F1 | F1 gain over the best single detector | +| --- | ---: | ---: | ---: | ---: | +| Silero, tuned threshold | 98.3 | 91.1 | 94.5 | | +| Ultra head, tuned threshold | 95.8 | 93.0 | 94.4 | | +| Redux head, tuned threshold | 96.9 | 92.8 | 94.8 | | +| Two-stage (Ultra) | 98.5 | 92.3 | 95.3 | +0.75 [+0.65, +0.86] | +| Two-stage (Redux) | 98.0 | 93.9 | 95.9 | +1.13 [+1.00, +1.28] | + +The gain is measured against the better of the two single detectors with their own tuned +threshold (Silero for Ultra, the Redux head for Redux). + +Recall when precision must stay at 99 percent or more (the threshold is chosen on the +training folds): + +| System | Recall at P of 99 percent or more | +| --- | ---: | +| Silero | 88.6 | +| Ultra head | 87.4 | +| Redux head | 89.4 | +| Two-stage (Ultra) | 90.6 | +| Two-stage (Redux) | 91.5 | + +### In plain language + +At the defaults on this corpus, for every 100 s of real speech Silero misses 14.4 s, +the Redux head 8.0 s and the two-stage rule 8.8 s (the Ultra head 10.6 s). For every +100 s that a detector calls speech, Silero is wrong for 0.4 s, the Redux head for 2.4 s +and the two-stage rule for 1.1 s (the Ultra head 1.4 s). The two-stage rule keeps most of +the head's extra recall and about two thirds of Silero's precision advantage over +the Redux head, and it never fires on noise-only audio. These are the default rows of the table +above, expressed in seconds; they are frame counts, not a statement about words. + +### Rules that did not help + +- **OR, max and mean of the two probabilities.** They inherit the head's false alarms on + noise. OR and max call 99.4 percent of the frames of the noise-only clips speech with + Ultra (97.8 percent with Redux), the same as the head alone. The mean at the default + weights is clean on noise-only clips (0.0 to 0.1 percent) but gained less on speech + (F1 93.4 for Redux against 94.9 for the two-stage rule). +- **AND and min.** No false alarms, but they are no better than Silero: recall 85.5 + against 85.6. +- **Switching between the two on an estimated noise level.** At the defaults it gave the + same false alarms on noise-only clips as the head alone (99.4 percent of the frames + with Ultra, 97.8 percent with Redux), so it did not remove the problem. +- **Logistic regression and gradient boosting on both probabilities and their neighbours.** + They scored the highest F1 on the synthetic clips (95.1 to 96.3 at threshold 0.5), but they + depend on the data they were fitted on. Fitted without noise-only clips, + boosting called 95 percent of the frames of noise-only clips speech with Ultra and 60 + percent with Redux, and the logistic regression 20 to 25 percent (Ultra) and 5 to 10 + percent (Redux). Adding noise-only clips to the training set brought those false alarms + down to under 0.3 percent, but the version fitted without them, applied to a real 600 s + talk, called about 90 percent of it speech, against 77.5 percent for Silero and 83 to + 84 percent for the heads (there was no reference for that talk, so this is a warning + sign and not a measured error). The two-stage rule called 84 percent. + +### Cost + +The two-stage rule needs both detectors to run. The cost is about the sum of the two. +From the speeds above (Silero about 660x real time and a head about 93x on one thread, +shared machine) both together run at about 80x real time. The rule itself is cheap +compared with the model runs, but it was not timed separately. + +### Caveats + +- Synthetic read speech with synthetic white and pink noise. Babble, music, + reverberation and non-English speech were not tested. +- Part of every tuned gain comes from the reference convention (it counts the pauses + inside an utterance as speech and uses an energy trim). A detector that bridges short + pauses gains from that. A gain of one F1 point is not a claim about real use. +- The word-time reference of the TED talks was lost, so the rules were not scored on + real talks. Only the share of speech per rule on one 600 s talk is reported. +- One corpus, one random seed for the bootstrap, one machine for the probabilities. + +Scripts, the small result files and how to reproduce: +[`scripts/vad_bench/fusion/`](../scripts/vad_bench/fusion/README.md). + ## When to use which This is limited to what the numbers above support. @@ -556,7 +933,11 @@ This is limited to what the numbers above support. MB, the command line tool needs about 60 MB, and it ran at about 660x real time on one thread on a shared machine. It gave frame level F1 of 91.7 to 94.4 on the synthetic data and 91.8 to 92.1 on the TED talks. -- **The Parakeet head when Ultra or Redux is already loaded.** The head costs no extra +- **Always-on gate, or audio with long stretches without speech.** Silero. The heads give + false alarms on noise-only audio, and the effect depends on the noise type and level (see + [the head on noise-only audio](#the-head-on-noise-only-audio)); Silero gave none. +- **The Parakeet head when Ultra or Redux is already loaded, on audio that is mostly + speech** (for example recorded talks before transcription). The head costs no extra model and its scores are close to Silero in the same tests: within about 1.5 F1 points on the synthetic clips at 5 dB and above (Ultra was 0.4 lower in white noise at 5 dB), 2 to 2.5 points higher on the TED talks, and 1.7 to 2.6 points higher at 0 dB pink @@ -575,8 +956,12 @@ This is limited to what the numbers above support. probabilities of parakeet.cpp match onnxruntime more closely (mean abs diff 0.00002, 2 flipped frames) than those of whisper.cpp do (0.00224, 309 flipped frames), but this did not change the segment scores. +- **Both together.** In an offline experiment, Silero deciding and the head only moving + the edges (the two-stage rule) gave 0.75 to 1.13 more F1 points than the best single + detector and no false alarms on noise. It is not implemented in parakeet.cpp. See + [the fusion experiment](#fusing-silero-and-the-head-offline-experiment). - **Not supported by these numbers:** any claim about GPUs, ARM, other languages, - music or non speech noise, streaming latency, or the effect of the VAD on WER. + music or non speech noise beyond the synthetic signals of [the noise study](#root-cause-study), streaming latency, or the effect of the VAD on WER. ## How to reproduce @@ -591,6 +976,7 @@ are committed. In short: 4. Silero parity: `make_clips.py`, build `wvad.cpp` against whisper.cpp, `silero_collect.py`, `silero_analyze.py`; speed with `speed_silero.py`. 5. Long talks: `longform_b1.sh`. +6. Noise root-cause study: [`noise_dive/`](../scripts/vad_bench/noise_dive/README.md). Small result files of the runs on this page (tables, per run timings and load logs) are in `scripts/vad_bench/results/`. The raw per clip predictions are not committed; diff --git a/docs/vad.md b/docs/vad.md index a8eda25..5388a08 100644 --- a/docs/vad.md +++ b/docs/vad.md @@ -15,6 +15,46 @@ transcription segments. The segmenter takes the frame period as a parameter Accuracy and speed numbers for both detectors, with the method and the scripts to repeat them, are in [vad-benchmarks.md](vad-benchmarks.md). +## Which detector to use + +The Parakeet VAD head alone gives false alarms on audio without speech. On a speech-free +file the head calls about 99 percent of the frames speech (Ultra 99.4 percent, Redux 97.8 +percent). Inside a file that has speech, over a 30 s stretch of noise at threshold 0.5, +the false-alarm frame rate was 17.7 percent for the Ultra head and 55 percent for the +Redux head. Silero showed 0 percent in both tests. This was measured on synthetic data: +LibriSpeech with added white and pink noise, see +[the experiment](vad-benchmarks.md#the-head-on-noise-only-audio). + +This is not a bug in parakeet.cpp. An independent reference built from Hugging Face +transformers and the documented head structure matches `parakeet-cli vad --probabilities` +to a maximum probability difference of 2e-4 on F16 files. It is a property of the model. +The head judges each 80 ms frame by its level and texture relative to the average of the +file it is in. Steady loud noise, or a file that is only noise, sits at a logit of about ++1.5 to +2 (probability 0.8 to 0.9). Speech sits at +5 to +12 and pauses at -3 to -8. The +cards of the model say the head exists so that recordings can be cut at pauses into +segments of at most 30 s. What it was trained on is not stated. It is not a noise or +music rejector. + +So use Silero, or Silero first with the head extending the boundaries, whenever long +stretches without speech are possible (always-on gates, calls, recordings with music or +noise). Use the head alone on audio that is known to be mostly speech (for example +before the transcription of recorded talks), or where its higher recall matters. If you +use the head alone and want fewer false alarms, raising the threshold to 0.9 removes most +noise false alarms at a cost of 1.5 to 2.6 F1 points on speech. Details, the noise types +that do and do not trigger it, and the mitigations we tested are in +[the benchmark page](vad-benchmarks.md#the-head-on-noise-only-audio). + +For `transcribe --vad` the user-visible cost is small: some wasted decoding, and rarely a +hallucinated phrase. The 30 s hard cut of the segmenter can also keep a large part of a +long noise gap, even where the head called none of it speech. This is a segmenter +behaviour, not a head false alarm, and it has not been fixed. + +An offline experiment also combined the two: Silero decides what is speech, and the head +only moves the edges. It scored 0.75 to 1.13 F1 points above the best single detector on +the synthetic clips and gave no false alarms on noise. It is not implemented in +parakeet.cpp. See +[Fusing Silero and the head](vad-benchmarks.md#fusing-silero-and-the-head-offline-experiment). + ## Standalone VAD API One call returns the speech regions of a clip as JSON, for either detector. Load diff --git a/scripts/vad_bench/fusion/README.md b/scripts/vad_bench/fusion/README.md new file mode 100644 index 0000000..3359307 --- /dev/null +++ b/scripts/vad_bench/fusion/README.md @@ -0,0 +1,70 @@ +# Silero plus head fusion experiment + +Offline experiment behind the section "Fusing Silero and the head (offline experiment)" +in [docs/vad-benchmarks.md](../../../docs/vad-benchmarks.md). It is not part of +parakeet.cpp: the fusion rules are a Python port of the segmenter, and no C++ code +implements them. + +No audio, no probability files and no pickles are committed. The four small result +files are: + +| File | What | +| --- | --- | +| `results.md` | All cross-validated tables for the Ultra head and the Redux head (precision, recall, F1 with bootstrap intervals, false alarms on noise-only clips, paired differences, noise-shift split, operating points at precision 99 percent or more) | +| `gap_results.md` | False alarms inside a 30 s noise stretch that sits in a file with speech. It is the printed output of `gapeval.py` with a longer heading | +| `pr_points.csv` | Points of the precision and recall frontiers (the data of the plot that `pr_curves.py` draws) | +| `ted_results.txt` | Output of `ted.py` on a 600 s talk: share of speech and agreement with Silero and with the head | + +## How it works + +Every script runs from one working directory (the "work directory") and reads and writes +files there with relative names: `data/`, `clips/`, `clips.json`, `gap.json`, +`probs.npz`, `probs_gap.npz`, `results.pkl`. All scripts need `numpy`, `soundfile`; +`run.py` and `gapeval.py` also need `scikit-learn`; `collect*.py` need `onnxruntime`; +`prep_libri.py` needs `datasets`; `pr_curves.py` needs `matplotlib`. Run them with +the work directory as the current directory, for example `python3 /path/to/fusion/run.py` +after copying or linking the scripts into it (the scripts import each other by name). + +The models are given by environment variables, read by `collect.py`, `collect_gap.py` +and `ted.py`: + +| Variable | Value | +| --- | --- | +| `SILERO_ONNX` | `silero_vad.onnx` of Silero VAD 6.2.3 | +| `PARAKEET_CLI` | built `parakeet-cli` (it must support `vad --probabilities`) | +| `ULTRA_GGUF` | ASR GGUF of `moondream/parakeet-ultra`, Q8_0 | +| `REDUX_GGUF` | ASR GGUF of `moondream/parakeet-redux`, packed (`keep`) | + +## Steps + +``` +python3 prep_libri.py # every 13th utterance of LibriSpeech test-clean -> data/libri.npy, data/spk.npy +python3 mkcorpus.py # 342 speech clips (38 sets of 5 utterances x 9 conditions) + 96 noise-only 30 s clips +python3 mkgap.py # 96 clips with a 30 s noise stretch between speech +python3 collect.py # per-frame probabilities of Silero and both heads -> probs.npz +python3 collect_gap.py # the same for the gap clips -> probs_gap.npz +python3 run.py # all rules, thresholds and the learned models -> results.pkl +python3 report.py # results.md (reads results.pkl) +python3 gapeval.py # prints the gap tables +python3 pr_curves.py # pr_curves.png and pr_points.csv +python3 detail.py # boundary error and missed segments -> detail.json +python3 ted.py talk.wav # speech share on one long talk (needs the clips and probs of the steps above) +``` + +The speaker folds are fixed by the seed in `mkcorpus.py`. `rules.py` holds the fusion +rules and the two-stage rule, `fl.py` the 10 ms grid, the post-processing and the +metrics, `analyze.py` the cross-validation and the bootstrap, `lib.py` the clip +builder and the noise. + +## Missing inputs + +- The TED talk used by `ted.py` is not part of this directory and is not fetched by + these scripts; `../fetch_data.py` fetches one TED-LIUM talk. The word-time reference of + the talk used in the main benchmark was lost, so the experiment could not score the + fusion rules against a TED reference. `ted.py` only reports how much of the talk each + rule calls speech and how well the rules agree with each other. +- `ted.py` fits its logistic regression on the synthetic clips, so it needs `clips.json`, + `clips/` and `probs.npz` from the steps above. +- `analyze.py` and `report.py` read `results.pkl` (about 50 MB), which is not committed. + Run `run.py` to make it. The random seeds are fixed, so a re-run should give the same tables + on the same probabilities, but this was not checked on a second machine. diff --git a/scripts/vad_bench/fusion/analyze.py b/scripts/vad_bench/fusion/analyze.py new file mode 100644 index 0000000..88a82c9 --- /dev/null +++ b/scripts/vad_bench/fusion/analyze.py @@ -0,0 +1,69 @@ +import pickle,json,sys,numpy as np +from fl import prf +R=pickle.load(open("results.pkl","rb")); CFG=R["cfg"]; LR=R["lr"]; M=R["meta"] +N=len(M["names"]); fold=np.array(M["fold"]); noz=np.array(M["noise_only"]); snr=np.array(M["snr_true"]); dur=np.array(M["dur"]) +sp=~noz +LEVELS=[("clean",99),("20 dB",20),("10 dB",10),("5 dB",5),("0 dB",0)] +rng=np.random.default_rng(0); B=2000 +def pooled(A,idx): c=A[idx,:4].sum(0); return prf(c) +def key(rule,p,h): return (rule,tuple(p),h,0.1) +def A_of(rule,p,h): return CFG[key(rule,p,"ultra" if rule=="silero" else h)] +HEADS=("ultra","redux") +# ---- candidate sets: dict name -> (list of params, array [ncfg,N,5]) ---- +def cand(rule,h): + hh="ultra" if rule=="silero" else h + ps=[k[1] for k in CFG if k[0]==rule and k[2]==hh and k[3]==0.1] + # keep insertion order + ps=[tuple(float(x) for x in p) for p in dict.fromkeys(ps)]; return ps,np.stack([CFG[key(rule,p,hh)] for p in ps]) +def lrcand(h,kind,split): o,c=LR[(h,kind,split)]; return [(float(t),) for t in __import__("rules").THR],o +def score(arr,idx,crit): + """arr [ncfg,N,5]; returns best cfg index on clips idx""" + c=arr[:,idx,:4].sum(1); tp,fp,fn=c[:,0],c[:,1],c[:,2] + P=tp/np.maximum(1,tp+fp);Rr=tp/np.maximum(1,tp+fn);F=2*P*Rr/np.maximum(1e-12,P+Rr) + if crit=="f1": return int(np.argmax(F)) + ok=P>=crit + return int(np.argmax(np.where(ok,Rr,-1+P*0)) if ok.any() else np.argmax(P)) +def crossfit(getarr,crit,mode="speaker"): + """returns (rows [N,5] with each clip from the cfg chosen without its own fold/shift side, chosen params per split)""" + rows=np.zeros((N,5)); chosen={} + if mode=="speaker": + for f in range(4): + ps,arr=getarr("f%d"%f); tr=np.flatnonzero(sp&(fold!=f)); te=np.flatnonzero(fold==f) + i=score(arr,tr,crit); rows[te]=arr[i][te]; chosen[f]=ps[i] + else: + ps,arr=getarr("shift"); tr=np.flatnonzero(sp&(snr>=10)); te=np.flatnonzero(snr<10) + i=score(arr,tr,crit); rows[te]=arr[i][te]; chosen["shift"]=ps[i] + return rows,chosen +def fixed(arr_ps,p): ps,arr=arr_ps; return arr[ps.index(tuple(p))] +def static(rule,h): + ps,arr=cand(rule,h); return lambda s:(ps,arr) +def lrget(h,kind): return lambda s:lrcand(h,kind,s) +# ---- bootstrap ---- +def boot_idx(idx): return rng.integers(0,len(idx),(B,len(idx))) +def ci(vals): return np.percentile(vals,[2.5,97.5]) +def prf_b(rows,idx,bi): + c=rows[idx][:,:4][bi].sum(1).astype(float) # B,4 + P=c[:,0]/np.maximum(1,c[:,0]+c[:,1]);Rr=c[:,0]/np.maximum(1,c[:,0]+c[:,2]);F=2*P*Rr/np.maximum(1e-12,P+Rr) + return P,Rr,F +def fmt(x,lo,hi): return f"{100*x:.1f} [{100*lo:.1f},{100*hi:.1f}]" +BI={} +def bidx(name,idx): + if name not in BI: BI[name]=(idx,rng.integers(0,len(idx),(B,len(idx)))) + return BI[name] +def level_idx(lv): return np.flatnonzero(sp&(snr==lv)) +def stats(rows): + """pooled + per level point estimates and CIs (paired resample indices fixed per index set)""" + out={} + for nm,idx in [("all",np.flatnonzero(sp))]+[(l,level_idx(v)) for l,v in LEVELS]: + idx,bi=bidx(nm,idx); P,Rr,F=prf_b(rows,idx,bi); p,r,f=pooled(rows,idx) + out[nm]=dict(P=p,R=r,F=f,Pci=ci(P),Rci=ci(Rr),Fci=ci(F),F_b=F) + return out +def fa(rows): + """false-alarm segments per hour and speech-frame FP rate on noise-only clips, per noise level""" + o={} + for l,v in LEVELS[1:]: + idx=np.flatnonzero(noz&(snr==v)); o[l]=(rows[idx,4].sum()/(dur[idx].sum()/3600), rows[idx,1].sum()/max(1,rows[idx,:4].sum())) + idx=np.flatnonzero(noz); o["all"]=(rows[idx,4].sum()/(dur[idx].sum()/3600), rows[idx,1].sum()/rows[idx,:4].sum()) + return o +if __name__=="__main__": + pass diff --git a/scripts/vad_bench/fusion/collect.py b/scripts/vad_bench/fusion/collect.py new file mode 100644 index 0000000..528721f --- /dev/null +++ b/scripts/vad_bench/fusion/collect.py @@ -0,0 +1,27 @@ +import json,os,subprocess,numpy as np,soundfile as sf,onnxruntime as ort +from multiprocessing import Pool +# Paths come from the environment: SILERO_ONNX (silero_vad.onnx, 6.2.3), PARAKEET_CLI (parakeet-cli), +# ULTRA_GGUF and REDUX_GGUF (the ASR GGUFs of the two heads). +M=os.environ["SILERO_ONNX"]; CLI=os.environ["PARAKEET_CLI"] +HEADS={"ultra":os.environ["ULTRA_GGUF"],"redux":os.environ["REDUX_GGUF"]} +meta=json.load(open("clips.json")) +def sil(y,sess): + n=(len(y)+511)//512; y=np.pad(y,(0,n*512-len(y))) + st=np.zeros((2,1,128),np.float32); ctx=np.zeros(64,np.float32); out=[] + for i in range(n): + x=np.concatenate([ctx,y[i*512:(i+1)*512]])[None].astype(np.float32) + o,st=sess.run(None,{"input":x,"state":st,"sr":np.array(16000,np.int64)}) + out.append(float(o[0,0])); ctx=x[0,-64:] + return np.array(out,np.float32) +def work(m): + so=ort.SessionOptions(); so.intra_op_num_threads=1; so.inter_op_num_threads=1 + sess=ort.InferenceSession(M,so,providers=["CPUExecutionProvider"]) + f=f"clips/{m['name']}.wav"; y=sf.read(f,dtype="float32")[0]; r={"silero":sil(y,sess)} + for k,g in HEADS.items(): + j=json.loads(subprocess.run([CLI,"vad","--model",g,"--input",f,"--probabilities","--threads","1"],capture_output=True,text=True,check=True).stdout) + r[k]=np.array(j["probabilities"],np.float32) + return m["name"],r +if __name__=="__main__": + with Pool(4) as p: res=p.map(work,meta,chunksize=4) + flat={f"{n}|{k}":v for n,r in res for k,v in r.items()} + np.savez_compressed("probs.npz",**flat); print("done",len(res)) diff --git a/scripts/vad_bench/fusion/collect_gap.py b/scripts/vad_bench/fusion/collect_gap.py new file mode 100644 index 0000000..b80ff58 --- /dev/null +++ b/scripts/vad_bench/fusion/collect_gap.py @@ -0,0 +1,27 @@ +import json,os,subprocess,numpy as np,soundfile as sf,onnxruntime as ort +from multiprocessing import Pool +# Paths come from the environment: SILERO_ONNX (silero_vad.onnx, 6.2.3), PARAKEET_CLI (parakeet-cli), +# ULTRA_GGUF and REDUX_GGUF (the ASR GGUFs of the two heads). +M=os.environ["SILERO_ONNX"]; CLI=os.environ["PARAKEET_CLI"] +HEADS={"ultra":os.environ["ULTRA_GGUF"],"redux":os.environ["REDUX_GGUF"]} +meta=json.load(open("gap.json")) +def sil(y,sess): + n=(len(y)+511)//512; y=np.pad(y,(0,n*512-len(y))) + st=np.zeros((2,1,128),np.float32); ctx=np.zeros(64,np.float32); out=[] + for i in range(n): + x=np.concatenate([ctx,y[i*512:(i+1)*512]])[None].astype(np.float32) + o,st=sess.run(None,{"input":x,"state":st,"sr":np.array(16000,np.int64)}) + out.append(float(o[0,0])); ctx=x[0,-64:] + return np.array(out,np.float32) +def work(m): + so=ort.SessionOptions(); so.intra_op_num_threads=1; so.inter_op_num_threads=1 + sess=ort.InferenceSession(M,so,providers=["CPUExecutionProvider"]) + f=f"clips/{m['name']}.wav"; y=sf.read(f,dtype="float32")[0]; r={"silero":sil(y,sess)} + for k,g in HEADS.items(): + j=json.loads(subprocess.run([CLI,"vad","--model",g,"--input",f,"--probabilities","--threads","1"],capture_output=True,text=True,check=True).stdout) + r[k]=np.array(j["probabilities"],np.float32) + return m["name"],r +if __name__=="__main__": + with Pool(4) as p: res=p.map(work,meta,chunksize=4) + flat={f"{n}|{k}":v for n,r in res for k,v in r.items()} + np.savez_compressed("probs_gap.npz",**flat); print("done",len(res)) diff --git a/scripts/vad_bench/fusion/detail.py b/scripts/vad_bench/fusion/detail.py new file mode 100644 index 0000000..6014283 --- /dev/null +++ b/scripts/vad_bench/fusion/detail.py @@ -0,0 +1,35 @@ +# Boundary error / missed segments / TED agreement for the cross-fitted systems +import json,numpy as np,soundfile as sf,subprocess +import run +from run import * +from analyze import crossfit,static,lrget,HEADS +folds=np.array([d["fold"] for d in D]); sp_idx=[i for i,d in enumerate(D) if not d["noise_only"]] +def build_systems(h): + S={} + for nm,(r,p) in {"Silero .5":("silero",(0.5,)),"head .5":("head",(0.5,)),"OR .5/.5":("or",(0.5,0.5)),"two-stage .5/.5/160/160/300":("two",(0.5,0.5,16,16,30))}.items(): + S[nm]=lambda d,f,r=r,p=p:raw_mask(r,p,d,h) + for r,nm in (("silero","Silero tuned"),("head","head tuned"),("or","OR tuned"),("and","AND tuned"),("mean","mean tuned"),("two","two-stage tuned")): + _,ch=crossfit(static(r,h),"f1"); S[nm]=lambda d,f,r=r,ch=ch:raw_mask(r,ch[f],d,h) + models={} + for kind in("lr6","hgb6","lr6nz","hgb6nz"): + base=kind.replace("nz",""); _,ch=crossfit(lrget(h,kind),"f1") + for f in range(4): + tr=[i for i in range(len(D)) if folds[i]!=f and (kind.endswith("nz") or not D[i]["noise_only"])] + X=np.concatenate([feats(D[i],h,base)[::10] for i in tr]); y=np.concatenate([D[i]["ref_mask"][::10] for i in tr]) + mdl=HistGradientBoostingClassifier(max_iter=120,learning_rate=0.1,random_state=0) if base=="hgb6" else LogisticRegression(C=1.0,max_iter=300) + models[(kind,f)]=mdl.fit(X,y) + S[f"{kind} tuned"]=lambda d,f,kind=kind,base=base,ch=ch:models[(kind,f)].predict_proba(feats(d,h,base))[:,1]>=ch[f][0] + return S +def metrics(fn,h): + st=[];en=[];miss=0;nref=0;nseg=0;cnt=np.zeros(4) + for i in sp_idx: + d=D[i]; m,regs=post(fn(d,d["fold"]),**POST_HEAD); a,b,ms=bnd(regs,d["ref"]); st+=a;en+=b;miss+=ms;nref+=len(d["ref"]);nseg+=len(regs); cnt+=counts(m,d["ref_mask"]) + q=lambda x:(np.median(x),np.percentile(x,90)) + return dict(start=q(st),end=q(en),miss=100*miss/nref,regions_per_ref=nseg/nref,PRF=[100*x for x in prf(cnt)]) +out={} +for h in HEADS: + out[h]={} + for nm,fn in build_systems(h).items(): + out[h][nm]=metrics(fn,h); o=out[h][nm] + print(f"{h:5s} {nm:30s} start {o['start'][0]:.0f}/{o['start'][1]:.0f} end {o['end'][0]:.0f}/{o['end'][1]:.0f} miss {o['miss']:.2f}% regions/ref {o['regions_per_ref']:.2f} PRF {o['PRF'][0]:.1f}/{o['PRF'][1]:.1f}/{o['PRF'][2]:.1f}",flush=True) +json.dump(out,open("detail.json","w"),default=float) diff --git a/scripts/vad_bench/fusion/fl.py b/scripts/vad_bench/fusion/fl.py new file mode 100644 index 0000000..f5f0050 --- /dev/null +++ b/scripts/vad_bench/fusion/fl.py @@ -0,0 +1,59 @@ +"""Fusion library: 10 ms grid, segmenter semantics, metrics.""" +import json,numpy as np,soundfile as sf +GR=0.01 +FS={"silero":0.032,"ultra":0.08,"redux":0.08} +def hold(p,fs,n): + """sample-and-hold: grid cell j (center (j+.5)*10ms) takes the value of the frame containing it; cells past the last frame get 0.""" + idx=((np.arange(n)+0.5)*GR/fs).astype(int) + out=np.zeros(n,np.float32); ok=idx0 and b1.0: miss+=1 # same convention as the earlier comparison: miss if a boundary is >1 s away + if ds<=1.0: st.append(ds*1000) + if de<=1.0: en.append(de*1000) + return st,en,miss +def load(name_filter=None): + meta=json.load(open("clips.json")); P=np.load("probs.npz") + D=[] + for m in meta: + n=int(round(m["dur"]/GR)); d=dict(m); d["n"]=n + d["ref_mask"]=np.zeros(n,bool) + for s,e in m["ref"]: d["ref_mask"][int(round(s/GR)):int(round(e/GR))]=True + for k in FS: d[k]=hold(P[f"{m['name']}|{k}"],FS[k],n) + y=sf.read(f"clips/{m['name']}.wav",dtype="float32")[0] + fr=len(y)//160; e2=(y[:fr*160].reshape(fr,160)**2).mean(1)+1e-12 + d["snr_est"]=10*np.log10(np.percentile(e2,90)/np.percentile(e2,10)) + D.append(d) + return D diff --git a/scripts/vad_bench/fusion/gap_results.md b/scripts/vad_bench/fusion/gap_results.md new file mode 100644 index 0000000..c488c2e --- /dev/null +++ b/scripts/vad_bench/fusion/gap_results.md @@ -0,0 +1,62 @@ +## ultra: FP frame rate inside embedded 30 s noise stretch % / false regions started in it (total over 96 clips: 12 per level x white,pink... 12 per level pooled over both noise types) +| system | 20 dB | 10 dB | 5 dB | 0 dB | all | +|---|---|---|---|---|---| +| Silero .5 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | +| head .5 | 0.0% / 0 | 0.6% / 19 | 14.3% / 183 | 55.8% / 450 | 17.7% / 652 | +| OR .5/.5 | 0.0% / 0 | 0.6% / 19 | 14.3% / 183 | 55.8% / 450 | 17.7% / 652 | +| AND .5/.5 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | +| mean .5/.5 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | +| max .5 | 0.0% / 0 | 0.6% / 19 | 14.3% / 183 | 55.8% / 450 | 17.7% / 652 | +| min .5 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | +| two-stage .5/.5/160/160/300 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | +| noise-switch .5/.5/T12 | 0.0% / 0 | 0.6% / 17 | 14.3% / 183 | 55.8% / 450 | 17.7% / 650 | +| Silero tuned | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | +| head tuned | 0.0% / 0 | 16.7% / 223 | 58.0% / 356 | 97.4% / 59 | 43.0% / 638 | +| OR tuned | 0.0% / 0 | 0.6% / 20 | 14.3% / 183 | 55.8% / 449 | 17.7% / 652 | +| AND tuned | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | +| max tuned | 0.0% / 0 | 11.9% / 205 | 50.9% / 359 | 95.6% / 88 | 39.6% / 652 | +| min tuned | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | +| mean tuned | 0.0% / 0 | 2.9% / 66 | 24.3% / 218 | 66.0% / 385 | 23.3% / 669 | +| two-stage tuned | 0.0% / 0 | 0.0% / 0 | 0.0% / 1 | 0.0% / 0 | 0.0% / 1 | +| noise-switch tuned | 0.0% / 0 | 8.7% / 151 | 44.9% / 335 | 92.5% / 133 | 36.5% / 619 | +| lr2 @.5 | 0.0% / 0 | 0.0% / 0 | 0.8% / 6 | 2.1% / 11 | 0.7% / 17 | +| lr2 tuned | 0.0% / 0 | 0.0% / 0 | 0.4% / 4 | 1.3% / 9 | 0.4% / 13 | +| lr6 @.5 | 0.0% / 0 | 0.0% / 0 | 0.5% / 3 | 1.7% / 4 | 0.5% / 7 | +| lr6 tuned | 0.0% / 0 | 0.0% / 0 | 0.2% / 1 | 1.1% / 5 | 0.3% / 6 | +| hgb6 @.5 | 0.0% / 0 | 0.2% / 8 | 3.9% / 92 | 17.8% / 303 | 5.5% / 403 | +| hgb6 tuned | 0.0% / 0 | 0.2% / 8 | 3.9% / 92 | 17.8% / 303 | 5.5% / 403 | +| lr6nz @.5 | 0.0% / 0 | 0.0% / 0 | 0.1% / 0 | 0.2% / 1 | 0.1% / 1 | +| lr6nz tuned | 0.0% / 0 | 0.0% / 0 | 0.1% / 1 | 0.2% / 1 | 0.1% / 2 | +| hgb6nz @.5 | 0.5% / 15 | 2.9% / 61 | 3.6% / 75 | 1.2% / 35 | 2.0% / 186 | +| hgb6nz tuned | 0.6% / 16 | 3.2% / 68 | 4.4% / 84 | 1.2% / 38 | 2.4% / 206 | +## redux: FP frame rate inside embedded 30 s noise stretch % / false regions started in it (total over 96 clips: 12 per level x white,pink... 12 per level pooled over both noise types) +| system | 20 dB | 10 dB | 5 dB | 0 dB | all | +|---|---|---|---|---|---| +| Silero .5 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | +| head .5 | 35.2% / 348 | 65.3% / 383 | 60.8% / 442 | 59.4% / 461 | 55.2% / 1634 | +| OR .5/.5 | 35.2% / 348 | 65.3% / 383 | 60.8% / 442 | 59.4% / 461 | 55.2% / 1634 | +| AND .5/.5 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | +| mean .5/.5 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | +| max .5 | 35.2% / 348 | 65.3% / 383 | 60.8% / 442 | 59.4% / 461 | 55.2% / 1634 | +| min .5 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | +| two-stage .5/.5/160/160/300 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | +| noise-switch .5/.5/T12 | 0.0% / 0 | 25.4% / 66 | 56.2% / 379 | 59.4% / 461 | 35.2% / 906 | +| Silero tuned | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | +| head tuned | 45.6% / 352 | 79.1% / 280 | 76.0% / 345 | 74.0% / 363 | 68.7% / 1340 | +| OR tuned | 16.9% / 260 | 35.0% / 414 | 29.2% / 437 | 33.0% / 432 | 28.5% / 1543 | +| AND tuned | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | +| max tuned | 45.6% / 352 | 79.1% / 280 | 76.0% / 345 | 74.0% / 363 | 68.7% / 1340 | +| min tuned | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | 0.0% / 0 | +| mean tuned | 17.2% / 262 | 35.5% / 416 | 30.1% / 438 | 34.0% / 433 | 29.2% / 1549 | +| two-stage tuned | 0.0% / 0 | 0.0% / 0 | 0.1% / 0 | 0.0% / 0 | 0.0% / 0 | +| noise-switch tuned | 35.2% / 348 | 65.3% / 383 | 60.8% / 442 | 59.4% / 461 | 55.2% / 1634 | +| lr2 @.5 | 0.4% / 1 | 0.7% / 4 | 0.9% / 6 | 1.3% / 7 | 0.8% / 18 | +| lr2 tuned | 0.1% / 0 | 0.2% / 2 | 0.5% / 3 | 0.8% / 6 | 0.4% / 11 | +| lr6 @.5 | 0.1% / 1 | 0.2% / 1 | 0.5% / 3 | 0.8% / 5 | 0.4% / 10 | +| lr6 tuned | 0.0% / 0 | 0.1% / 1 | 0.3% / 0 | 0.3% / 2 | 0.2% / 3 | +| hgb6 @.5 | 1.9% / 44 | 3.3% / 76 | 2.5% / 54 | 5.1% / 102 | 3.2% / 276 | +| hgb6 tuned | 2.2% / 51 | 3.4% / 77 | 2.6% / 58 | 5.4% / 105 | 3.4% / 291 | +| lr6nz @.5 | 0.0% / 0 | 0.0% / 0 | 0.1% / 0 | 0.1% / 0 | 0.1% / 0 | +| lr6nz tuned | 0.0% / 0 | 0.0% / 0 | 0.1% / 0 | 0.1% / 0 | 0.1% / 0 | +| hgb6nz @.5 | 0.9% / 15 | 0.2% / 8 | 0.4% / 8 | 0.4% / 11 | 0.5% / 42 | +| hgb6nz tuned | 1.0% / 15 | 0.2% / 8 | 0.4% / 8 | 0.4% / 12 | 0.5% / 43 | \ No newline at end of file diff --git a/scripts/vad_bench/fusion/gapeval.py b/scripts/vad_bench/fusion/gapeval.py new file mode 100644 index 0000000..687799d --- /dev/null +++ b/scripts/vad_bench/fusion/gapeval.py @@ -0,0 +1,51 @@ +# False alarms inside a 30 s speech-free noise stretch embedded in a file that also has speech +import json,numpy as np,soundfile as sf +import run +from run import * +from analyze import crossfit,static,lrget,HEADS +G=json.load(open("gap.json")); PG=np.load("probs_gap.npz"); GD=[] +for m in G: + n=int(round(m["dur"]/GR)); d=dict(m); d["n"]=n; d["ref_mask"]=np.zeros(n,bool) + for s,e in m["ref"]: d["ref_mask"][int(round(s/GR)):int(round(e/GR))]=True + for k in FS: d[k]=hold(PG[f"{m['name']}|{k}"],FS[k],n) + y=sf.read(f"clips/{m['name']}.wav",dtype="float32")[0]; fr=len(y)//160; e2=(y[:fr*160].reshape(fr,160)**2).mean(1)+1e-12 + d["snr_est"]=10*np.log10(np.percentile(e2,90)/np.percentile(e2,10)); d["snr_true"]=int(m["cond"].lstrip("whitepink")); GD.append(d) +folds=np.array([d["fold"] for d in D]); easy=None +def fa_stretch(mask,d): + m,regs=post(mask,**POST_HEAD); a,b=d["stretch"]; ia,ib=int(a/GR),int(b/GR) + fp=m[ia:ib].mean(); nseg=sum(1 for s,e in regs if a<=s=(thr if thr is not None else ch[f][0]) + res={} + for nm,fn in sysmask.items(): + L={}; + for d in GD: + fp,ns=fa_stretch(fn(d,d["fold"]),d); L.setdefault(d["snr_true"],[]).append((fp,ns)) + res[nm]={snr_:(float(np.mean([x[0] for x in v])),float(np.sum([x[1] for x in v]))) for snr_,v in L.items()} + allv=[x for v in L.values() for x in v]; res[nm]["all"]=(float(np.mean([x[0] for x in allv])),float(np.sum([x[1] for x in allv]))) + out[h]=res +json.dump(out,open("gap_results.json","w")) +for h,res in out.items(): + print(f"## {h}: embedded 30 s noise stretch ({len(GD)} clips = 12 per level x2 noise types... see below), FP frame rate in the stretch % / false regions started in stretch (total over clips)") + print("| system | 20 dB | 10 dB | 5 dB | 0 dB | all |"); print("|---|---|---|---|---|---|") + for nm,r in res.items(): print(f"| {nm} | "+" | ".join(f"{100*r[k][0]:.1f}% / {r[k][1]:.0f}" for k in (20,10,5,0,"all"))+" |") diff --git a/scripts/vad_bench/fusion/lib.py b/scripts/vad_bench/fusion/lib.py new file mode 100644 index 0000000..b27114c --- /dev/null +++ b/scripts/vad_bench/fusion/lib.py @@ -0,0 +1,46 @@ +import numpy as np +GR=0.01 # mask grid 10 ms +def mask(spans,n_sec): + m=np.zeros(int(round(n_sec/GR)),bool) + for s,e in spans: + m[int(round(s/GR)):int(round(e/GR))]=True + return m +def trim(u): + fr=len(u)//160; r=np.sqrt((u[:fr*160].reshape(fr,160)**2).mean(1)) + idx=np.where(r>r.max()*0.01)[0] + return u[idx[0]*160:(idx[-1]+1)*160] +def pink(n,rng): + X=np.fft.rfft(rng.standard_normal(n)); f=np.arange(len(X)); f[0]=1 + y=np.fft.irfft(X/np.sqrt(f),n); return y/y.std() +def build(utts,rng): + g=lambda a,b: rng.uniform(a,b) + parts=[];spans=[];t=g(0.5,1.5) + parts.append(rng.standard_normal(int(t*16000))*1e-3) + for i,u in enumerate(utts): + u=trim(u); spans.append((t,t+len(u)/16000)); parts.append(u); t+=len(u)/16000 + gap=g(0.3,2.0) if i1.0: miss+=1 + else: out.append(d*1000) + return out,miss + diff --git a/scripts/vad_bench/fusion/mkcorpus.py b/scripts/vad_bench/fusion/mkcorpus.py new file mode 100644 index 0000000..4c527d3 --- /dev/null +++ b/scripts/vad_bench/fusion/mkcorpus.py @@ -0,0 +1,29 @@ +# Build the fusion corpus: 4 speaker folds x 10 clips x 9 conditions (+ noise-only clips). Speakers never cross folds. +import json,os,numpy as np,soundfile as sf +from lib import * +libri=list(np.load("data/libri.npy",allow_pickle=True)); spk=np.load("data/spk.npy") +rng=np.random.default_rng(11) +sp_ids=rng.permutation(sorted(set(spk))); fold_of={s:i%4 for i,s in enumerate(sp_ids)} +sets=[] # (fold, utts) +for f in range(4): + idx=[i for i in range(len(libri)) if fold_of[spk[i]]==f]; idx=list(rng.permutation(idx)) + for k in range(min(10,len(idx)//5)): sets.append((f,[libri[i] for i in idx[k*5:(k+1)*5]])) +conds=[("clean",None)]+[(k,s) for k in("white","pink") for s in(20,10,5,0)] +os.makedirs("clips",exist_ok=True); meta=[]; Ps=[] +for ci,(kind,snr) in enumerate(conds): + for k,(f,us) in enumerate(sets): + y,spans=build(us,np.random.default_rng(1000+k)) + if ci==0: Ps.append(float(np.mean(np.concatenate([y[int(s*16000):int(e*16000)] for s,e in spans])**2))) + y=add_noise(y,spans,kind,snr,np.random.default_rng(5000+ci*100+k)) + name=f"{kind}{'' if snr is None else snr}_{k:02d}" + sf.write(f"clips/{name}.wav",y,16000,subtype="PCM_16") + meta.append(dict(name=name,cond=name.split("_")[0],fold=f,ref=spans,dur=len(y)/16000)) +# noise-only: 12 x 30 s per noise condition, level from a speech set's power (same SNR definition) +for ci,(kind,snr) in enumerate(conds[1:]): + for k in range(12): + r=np.random.default_rng(9000+ci*100+k); P=Ps[(k*3)%len(Ps)]; n=30*16000 + z=r.standard_normal(n) if kind=="white" else pink(n,r) + z=np.clip(z*np.sqrt(P/10**(snr/10)),-1,1).astype(np.float32) + name=f"nz-{kind}{snr}_{k:02d}"; sf.write(f"clips/{name}.wav",z,16000,subtype="PCM_16") + meta.append(dict(name=name,cond=f"nz-{kind}{snr}",fold=-1,ref=[],dur=30.0)) +json.dump(meta,open("clips.json","w")); print(len(meta),sum(m["dur"] for m in meta)) diff --git a/scripts/vad_bench/fusion/mkgap.py b/scripts/vad_bench/fusion/mkgap.py new file mode 100644 index 0000000..894e912 --- /dev/null +++ b/scripts/vad_bench/fusion/mkgap.py @@ -0,0 +1,21 @@ +# "Embedded noise stretch" clips: 2 utterances, 30 s of noise-only, 3 utterances; noise over the whole clip at the given SNR. +# Tests false alarms where the file also contains speech (the head normalises features per file). +import json,numpy as np,soundfile as sf +from lib import * +libri=list(np.load("data/libri.npy",allow_pickle=True)); spk=np.load("data/spk.npy") +rng=np.random.default_rng(11) +sp_ids=rng.permutation(sorted(set(spk))); fold_of={s:i%4 for i,s in enumerate(sp_ids)} +meta=[] +conds=[(k,s) for k in("white","pink") for s in(20,10,5,0)] +for ci,(kind,snr) in enumerate(conds): + for k in range(12): + f=k%4; idx=[i for i in range(len(libri)) if fold_of[spk[i]]==f]; r=np.random.default_rng(300+ci*50+k); idx=list(r.permutation(idx))[:5] + us=[libri[i] for i in idx] + y1,s1=build(us[:2],np.random.default_rng(400+k)); y2,s2=build(us[2:],np.random.default_rng(500+k)) + off=len(y1)/16000+30.0 + y=np.concatenate([y1,np.random.default_rng(600+k).standard_normal(30*16000)*1e-3,y2]).astype(np.float32) + spans=s1+[(a+off,b+off) for a,b in s2] + y=add_noise(y,spans,kind,snr,np.random.default_rng(7000+ci*100+k)) + name=f"gap-{kind}{snr}_{k:02d}"; sf.write(f"clips/{name}.wav",y,16000,subtype="PCM_16") + meta.append(dict(name=name,cond=f"{kind}{snr}",fold=f,ref=spans,dur=len(y)/16000,stretch=[s1[-1][1]+0.5,s2[0][0]+off-0.5])) +json.dump(meta,open("gap.json","w")); print(len(meta),sum(m["dur"] for m in meta)) diff --git a/scripts/vad_bench/fusion/pr_curves.py b/scripts/vad_bench/fusion/pr_curves.py new file mode 100644 index 0000000..da61234 --- /dev/null +++ b/scripts/vad_bench/fusion/pr_curves.py @@ -0,0 +1,26 @@ +from analyze import * +import matplotlib; matplotlib.use("Agg"); import matplotlib.pyplot as plt +allidx=np.flatnonzero(sp); rows=[] +def pr(arr): c=arr[:,allidx,:4].sum(1); return c[:,0]/(c[:,0]+c[:,1]),c[:,0]/(c[:,0]+c[:,2]) +def front(P,Rr): + o=np.argsort(-Rr); best=-1; keep=[] + for i in o: + if P[i]>best+1e-9: keep.append(i); best=P[i] + return keep +fig,axs=plt.subplots(1,2,figsize=(13,5.5)) +for ax,h in zip(axs,HEADS): + for r_,lab,st in (("silero","Silero thr sweep","k-"),("head",f"{h} head thr sweep","r-"),("or","OR grid frontier","g--"),("and","AND grid frontier","c--"),("mean","mean (w .25/.5/.75) frontier","m--"),("max","max sweep","y:"),("two","two-stage frontier","b-")): + ps,arr=cand(r_,h); P,Rr=pr(arr); k=front(P,Rr); ax.plot(100*Rr[k],100*P[k],st,label=lab,lw=1.6) + for i in k: rows.append((h,r_,str(ps[i]),round(100*P[i],2),round(100*Rr[i],2))) + # LR out-of-fold curves + for kind,st in (("lr6","orange"),("hgb6","brown"),("lr6nz","olive")): + P=[];Rr=[] + for ti in range(17): + c=np.zeros(4) + for f in range(4): o,_=LR[(h,kind,"f%d"%f)]; te=np.flatnonzero(sp&(fold==f)); c+=o[ti][te,:4].sum(0) + P.append(c[0]/(c[0]+c[1])); Rr.append(c[0]/(c[0]+c[2])) + P=np.array(P);Rr=np.array(Rr); ax.plot(100*Rr,100*P,"-",color=st,label=f"{kind} out-of-fold thr sweep",lw=1.6) + for ti in range(17): rows.append((h,kind+"_oof",f"thr={0.1+0.05*ti:.2f}",round(100*P[ti],2),round(100*Rr[ti],2))) + ax.set_xlim(80,100); ax.set_ylim(93,100); ax.set_xlabel("frame recall %"); ax.set_ylabel("frame precision %"); ax.set_title(f"Silero + {h} head (342 synthetic speech clips)"); ax.grid(alpha=.3); ax.legend(fontsize=7,loc="lower left") +plt.tight_layout(); plt.savefig("pr_curves.png",dpi=130) +import csv; w=csv.writer(open("pr_points.csv","w")); w.writerow(["head","rule","params","precision_pct","recall_pct"]); w.writerows(rows) diff --git a/scripts/vad_bench/fusion/pr_points.csv b/scripts/vad_bench/fusion/pr_points.csv new file mode 100644 index 0000000..daee549 --- /dev/null +++ b/scripts/vad_bench/fusion/pr_points.csv @@ -0,0 +1,546 @@ +head,rule,params,precision_pct,recall_pct +ultra,silero,"(0.1,)",98.29,91.05 +ultra,silero,"(0.15,)",98.82,89.76 +ultra,silero,"(0.2,)",99.07,88.84 +ultra,silero,"(0.25,)",99.24,88.14 +ultra,silero,"(0.3,)",99.34,87.54 +ultra,silero,"(0.35,)",99.43,86.98 +ultra,silero,"(0.4,)",99.5,86.49 +ultra,silero,"(0.45,)",99.56,86.02 +ultra,silero,"(0.5,)",99.61,85.56 +ultra,silero,"(0.55,)",99.64,85.13 +ultra,silero,"(0.6,)",99.68,84.71 +ultra,silero,"(0.65,)",99.71,84.25 +ultra,silero,"(0.7,)",99.74,83.78 +ultra,silero,"(0.75,)",99.76,83.28 +ultra,silero,"(0.8,)",99.8,82.68 +ultra,silero,"(0.85,)",99.83,81.97 +ultra,silero,"(0.9,)",99.86,80.88 +ultra,head,"(0.1,)",91.92,96.24 +ultra,head,"(0.15,)",93.56,95.06 +ultra,head,"(0.2,)",94.79,94.09 +ultra,head,"(0.25,)",95.83,93.18 +ultra,head,"(0.3,)",96.65,92.33 +ultra,head,"(0.35,)",97.33,91.54 +ultra,head,"(0.4,)",97.85,90.79 +ultra,head,"(0.45,)",98.26,90.09 +ultra,head,"(0.5,)",98.58,89.4 +ultra,head,"(0.55,)",98.82,88.61 +ultra,head,"(0.6,)",99.0,87.76 +ultra,head,"(0.65,)",99.15,86.87 +ultra,head,"(0.7,)",99.28,85.91 +ultra,head,"(0.75,)",99.36,84.84 +ultra,head,"(0.8,)",99.47,83.46 +ultra,head,"(0.85,)",99.53,81.77 +ultra,head,"(0.9,)",99.6,79.05 +ultra,or,"(0.1, 0.1)",91.75,96.66 +ultra,or,"(0.2, 0.1)",91.89,96.37 +ultra,or,"(0.3, 0.1)",91.91,96.3 +ultra,or,"(0.4, 0.1)",91.92,96.27 +ultra,or,"(0.5, 0.1)",91.92,96.26 +ultra,or,"(0.1, 0.2)",94.33,95.18 +ultra,or,"(0.2, 0.2)",94.66,94.57 +ultra,or,"(0.3, 0.2)",94.75,94.34 +ultra,or,"(0.1, 0.3)",95.97,94.27 +ultra,or,"(0.1, 0.4)",97.01,93.55 +ultra,or,"(0.1, 0.5)",97.59,93.07 +ultra,or,"(0.1, 0.6)",97.9,92.64 +ultra,or,"(0.1, 0.7)",98.09,92.26 +ultra,or,"(0.1, 0.8)",98.18,91.92 +ultra,or,"(0.2, 0.5)",98.22,91.71 +ultra,or,"(0.1, 0.9)",98.23,91.54 +ultra,or,"(0.2, 0.6)",98.58,91.12 +ultra,or,"(0.2, 0.7)",98.81,90.55 +ultra,or,"(0.2, 0.8)",98.92,90.09 +ultra,or,"(0.3, 0.7)",99.03,89.68 +ultra,or,"(0.4, 0.7)",99.14,89.11 +ultra,or,"(0.3, 0.8)",99.16,89.09 +ultra,or,"(0.5, 0.7)",99.2,88.67 +ultra,or,"(0.4, 0.8)",99.27,88.43 +ultra,or,"(0.5, 0.8)",99.35,87.89 +ultra,or,"(0.4, 0.9)",99.37,87.6 +ultra,or,"(0.6, 0.8)",99.39,87.46 +ultra,or,"(0.7, 0.8)",99.42,87.05 +ultra,or,"(0.5, 0.9)",99.45,86.94 +ultra,or,"(0.6, 0.9)",99.5,86.37 +ultra,or,"(0.7, 0.9)",99.54,85.8 +ultra,or,"(0.8, 0.9)",99.57,85.21 +ultra,or,"(0.9, 0.9)",99.59,84.38 +ultra,and,"(0.1, 0.1)",98.52,90.51 +ultra,and,"(0.1, 0.2)",98.8,89.78 +ultra,and,"(0.1, 0.3)",99.02,88.93 +ultra,and,"(0.2, 0.1)",99.12,88.67 +ultra,and,"(0.2, 0.2)",99.22,88.23 +ultra,and,"(0.2, 0.3)",99.31,87.7 +ultra,and,"(0.3, 0.1)",99.36,87.45 +ultra,and,"(0.3, 0.2)",99.4,87.18 +ultra,and,"(0.3, 0.3)",99.45,86.82 +ultra,and,"(0.4, 0.1)",99.5,86.45 +ultra,and,"(0.4, 0.2)",99.52,86.28 +ultra,and,"(0.4, 0.3)",99.55,85.99 +ultra,and,"(0.4, 0.4)",99.57,85.58 +ultra,and,"(0.5, 0.1)",99.61,85.54 +ultra,and,"(0.5, 0.2)",99.62,85.41 +ultra,and,"(0.5, 0.3)",99.63,85.19 +ultra,and,"(0.5, 0.4)",99.65,84.85 +ultra,and,"(0.6, 0.1)",99.68,84.68 +ultra,and,"(0.6, 0.2)",99.68,84.6 +ultra,and,"(0.6, 0.3)",99.69,84.4 +ultra,and,"(0.6, 0.4)",99.7,84.12 +ultra,and,"(0.7, 0.1)",99.74,83.76 +ultra,and,"(0.7, 0.2)",99.74,83.69 +ultra,and,"(0.7, 0.3)",99.75,83.51 +ultra,and,"(0.7, 0.4)",99.75,83.29 +ultra,and,"(0.7, 0.5)",99.76,82.95 +ultra,and,"(0.8, 0.1)",99.8,82.66 +ultra,and,"(0.8, 0.2)",99.8,82.59 +ultra,and,"(0.8, 0.3)",99.8,82.43 +ultra,and,"(0.8, 0.4)",99.8,82.25 +ultra,and,"(0.8, 0.5)",99.81,81.95 +ultra,and,"(0.8, 0.6)",99.82,81.44 +ultra,and,"(0.9, 0.1)",99.86,80.88 +ultra,and,"(0.9, 0.2)",99.86,80.82 +ultra,and,"(0.9, 0.3)",99.86,80.7 +ultra,and,"(0.9, 0.4)",99.86,80.55 +ultra,and,"(0.9, 0.5)",99.86,80.31 +ultra,and,"(0.9, 0.6)",99.87,79.87 +ultra,and,"(0.9, 0.7)",99.87,79.23 +ultra,and,"(0.9, 0.8)",99.88,78.03 +ultra,and,"(0.9, 0.9)",99.89,75.26 +ultra,mean,"(0.25, 0.1)",92.66,95.97 +ultra,mean,"(0.5, 0.1)",93.82,95.43 +ultra,mean,"(0.25, 0.15)",94.41,94.7 +ultra,mean,"(0.75, 0.1)",96.13,94.28 +ultra,mean,"(0.5, 0.2)",97.3,92.75 +ultra,mean,"(0.75, 0.15)",97.91,92.49 +ultra,mean,"(0.5, 0.25)",98.11,91.7 +ultra,mean,"(0.25, 0.35)",98.21,91.04 +ultra,mean,"(0.75, 0.2)",98.66,91.02 +ultra,mean,"(0.5, 0.35)",98.97,89.92 +ultra,mean,"(0.75, 0.25)",99.05,89.63 +ultra,mean,"(0.5, 0.4)",99.18,89.1 +ultra,mean,"(0.75, 0.3)",99.24,88.49 +ultra,mean,"(0.5, 0.45)",99.31,88.16 +ultra,mean,"(0.75, 0.35)",99.35,87.73 +ultra,mean,"(0.75, 0.4)",99.44,87.14 +ultra,mean,"(0.5, 0.5)",99.45,87.12 +ultra,mean,"(0.75, 0.45)",99.51,86.57 +ultra,mean,"(0.5, 0.55)",99.53,86.27 +ultra,mean,"(0.75, 0.5)",99.56,86.06 +ultra,mean,"(0.75, 0.55)",99.61,85.54 +ultra,mean,"(0.75, 0.6)",99.66,85.03 +ultra,mean,"(0.75, 0.65)",99.69,84.52 +ultra,mean,"(0.5, 0.7)",99.69,83.96 +ultra,mean,"(0.75, 0.7)",99.73,83.93 +ultra,mean,"(0.75, 0.75)",99.76,83.31 +ultra,mean,"(0.75, 0.8)",99.79,82.48 +ultra,mean,"(0.75, 0.85)",99.83,81.38 +ultra,mean,"(0.75, 0.9)",99.86,79.61 +ultra,max,"(0.1,)",91.75,96.66 +ultra,max,"(0.15,)",93.44,95.47 +ultra,max,"(0.2,)",94.66,94.57 +ultra,max,"(0.25,)",95.73,93.73 +ultra,max,"(0.3,)",96.57,92.95 +ultra,max,"(0.35,)",97.26,92.23 +ultra,max,"(0.4,)",97.8,91.57 +ultra,max,"(0.45,)",98.2,91.04 +ultra,max,"(0.5,)",98.54,90.48 +ultra,max,"(0.55,)",98.78,89.9 +ultra,max,"(0.6,)",98.96,89.29 +ultra,max,"(0.65,)",99.11,88.66 +ultra,max,"(0.7,)",99.25,88.03 +ultra,max,"(0.75,)",99.33,87.39 +ultra,max,"(0.8,)",99.44,86.62 +ultra,max,"(0.85,)",99.51,85.7 +ultra,max,"(0.9,)",99.59,84.38 +ultra,two,"(0.3, 0.3, 48.0, 32.0, 60.0)",98.3,92.39 +ultra,two,"(0.3, 0.3, 48.0, 32.0, 30.0)",98.32,92.37 +ultra,two,"(0.3, 0.3, 48.0, 32.0, 15.0)",98.33,92.37 +ultra,two,"(0.3, 0.3, 32.0, 32.0, 60.0)",98.36,92.36 +ultra,two,"(0.3, 0.3, 48.0, 16.0, 60.0)",98.39,92.34 +ultra,two,"(0.3, 0.3, 32.0, 32.0, 0.0)",98.39,92.32 +ultra,two,"(0.3, 0.3, 32.0, 16.0, 60.0)",98.46,92.31 +ultra,two,"(0.3, 0.3, 32.0, 16.0, 30.0)",98.48,92.25 +ultra,two,"(0.3, 0.3, 32.0, 16.0, 0.0)",98.49,92.24 +ultra,two,"(0.3, 0.3, 48.0, 8.0, 60.0)",98.5,92.18 +ultra,two,"(0.3, 0.3, 16.0, 16.0, 60.0)",98.53,92.15 +ultra,two,"(0.3, 0.3, 32.0, 8.0, 60.0)",98.57,92.14 +ultra,two,"(0.3, 0.3, 32.0, 8.0, 30.0)",98.6,92.04 +ultra,two,"(0.3, 0.3, 32.0, 8.0, 0.0)",98.6,92.01 +ultra,two,"(0.3, 0.3, 16.0, 8.0, 60.0)",98.65,91.98 +ultra,two,"(0.3, 0.3, 16.0, 8.0, 30.0)",98.68,91.84 +ultra,two,"(0.3, 0.3, 16.0, 8.0, 15.0)",98.68,91.78 +ultra,two,"(0.5, 0.3, 16.0, 16.0, 60.0)",98.7,91.66 +ultra,two,"(0.5, 0.3, 16.0, 16.0, 30.0)",98.72,91.52 +ultra,two,"(0.5, 0.3, 48.0, 8.0, 60.0)",98.73,91.51 +ultra,two,"(0.3, 0.3, 8.0, 8.0, 60.0)",98.79,91.48 +ultra,two,"(0.5, 0.3, 32.0, 8.0, 60.0)",98.79,91.45 +ultra,two,"(0.5, 0.3, 32.0, 8.0, 30.0)",98.8,91.32 +ultra,two,"(0.5, 0.3, 16.0, 8.0, 60.0)",98.86,91.28 +ultra,two,"(0.5, 0.3, 16.0, 8.0, 30.0)",98.88,91.07 +ultra,two,"(0.5, 0.3, 16.0, 8.0, 15.0)",98.89,90.99 +ultra,two,"(0.3, 0.3, 16.0, 0.0, 60.0)",98.94,90.98 +ultra,two,"(0.3, 0.3, 16.0, 0.0, 30.0)",98.96,90.78 +ultra,two,"(0.5, 0.3, 8.0, 8.0, 60.0)",99.01,90.64 +ultra,two,"(0.3, 0.3, 8.0, 0.0, 60.0)",99.08,90.51 +ultra,two,"(0.3, 0.3, 8.0, 0.0, 30.0)",99.1,90.12 +ultra,two,"(0.5, 0.3, 32.0, 0.0, 60.0)",99.12,90.05 +ultra,two,"(0.7, 0.3, 8.0, 8.0, 60.0)",99.14,89.92 +ultra,two,"(0.5, 0.3, 16.0, 0.0, 60.0)",99.21,89.85 +ultra,two,"(0.5, 0.3, 16.0, 0.0, 30.0)",99.23,89.51 +ultra,two,"(0.5, 0.3, 8.0, 0.0, 60.0)",99.36,89.35 +ultra,two,"(0.5, 0.3, 8.0, 0.0, 30.0)",99.38,88.73 +ultra,two,"(0.7, 0.3, 8.0, 0.0, 60.0)",99.51,88.48 +ultra,two,"(0.5, 0.3, 0.0, 0.0, 60.0)",99.6,88.39 +ultra,two,"(0.7, 0.3, 0.0, 0.0, 60.0)",99.72,87.54 +ultra,two,"(0.7, 0.5, 0.0, 0.0, 60.0)",99.74,86.07 +ultra,two,"(0.7, 0.3, 0.0, 0.0, 30.0)",99.74,85.13 +ultra,lr6_oof,thr=0.10,84.63,99.84 +ultra,lr6_oof,thr=0.15,85.91,99.48 +ultra,lr6_oof,thr=0.20,87.46,99.0 +ultra,lr6_oof,thr=0.25,89.17,98.48 +ultra,lr6_oof,thr=0.30,90.88,97.93 +ultra,lr6_oof,thr=0.35,92.4,97.34 +ultra,lr6_oof,thr=0.40,93.79,96.71 +ultra,lr6_oof,thr=0.45,94.96,96.07 +ultra,lr6_oof,thr=0.50,95.93,95.36 +ultra,lr6_oof,thr=0.55,96.75,94.68 +ultra,lr6_oof,thr=0.60,97.44,93.93 +ultra,lr6_oof,thr=0.65,98.0,93.13 +ultra,lr6_oof,thr=0.70,98.44,92.3 +ultra,lr6_oof,thr=0.75,98.79,91.38 +ultra,lr6_oof,thr=0.80,99.08,90.29 +ultra,lr6_oof,thr=0.85,99.31,89.03 +ultra,lr6_oof,thr=0.90,99.53,87.43 +ultra,hgb6_oof,thr=0.10,84.13,99.99 +ultra,hgb6_oof,thr=0.15,86.31,99.68 +ultra,hgb6_oof,thr=0.20,89.79,98.93 +ultra,hgb6_oof,thr=0.25,91.97,98.3 +ultra,hgb6_oof,thr=0.30,93.7,97.7 +ultra,hgb6_oof,thr=0.35,94.9,97.07 +ultra,hgb6_oof,thr=0.40,95.74,96.65 +ultra,hgb6_oof,thr=0.45,96.26,96.24 +ultra,hgb6_oof,thr=0.50,96.72,95.83 +ultra,hgb6_oof,thr=0.55,97.1,95.33 +ultra,hgb6_oof,thr=0.60,97.42,94.84 +ultra,hgb6_oof,thr=0.65,97.74,94.33 +ultra,hgb6_oof,thr=0.70,98.06,93.73 +ultra,hgb6_oof,thr=0.75,98.44,92.78 +ultra,hgb6_oof,thr=0.80,98.81,91.65 +ultra,hgb6_oof,thr=0.85,99.15,90.39 +ultra,hgb6_oof,thr=0.90,99.43,88.99 +ultra,lr6nz_oof,thr=0.10,85.49,99.47 +ultra,lr6nz_oof,thr=0.15,86.93,99.04 +ultra,lr6nz_oof,thr=0.20,88.61,98.51 +ultra,lr6nz_oof,thr=0.25,90.39,97.87 +ultra,lr6nz_oof,thr=0.30,92.06,97.27 +ultra,lr6nz_oof,thr=0.35,93.52,96.63 +ultra,lr6nz_oof,thr=0.40,94.76,96.0 +ultra,lr6nz_oof,thr=0.45,95.78,95.33 +ultra,lr6nz_oof,thr=0.50,96.61,94.66 +ultra,lr6nz_oof,thr=0.55,97.3,94.03 +ultra,lr6nz_oof,thr=0.60,97.82,93.3 +ultra,lr6nz_oof,thr=0.65,98.21,92.59 +ultra,lr6nz_oof,thr=0.70,98.52,91.81 +ultra,lr6nz_oof,thr=0.75,98.81,90.97 +ultra,lr6nz_oof,thr=0.80,99.07,90.01 +ultra,lr6nz_oof,thr=0.85,99.29,88.88 +ultra,lr6nz_oof,thr=0.90,99.5,87.48 +redux,silero,"(0.1,)",98.29,91.05 +redux,silero,"(0.15,)",98.82,89.76 +redux,silero,"(0.2,)",99.07,88.84 +redux,silero,"(0.25,)",99.24,88.14 +redux,silero,"(0.3,)",99.34,87.54 +redux,silero,"(0.35,)",99.43,86.98 +redux,silero,"(0.4,)",99.5,86.49 +redux,silero,"(0.45,)",99.56,86.02 +redux,silero,"(0.5,)",99.61,85.56 +redux,silero,"(0.55,)",99.64,85.13 +redux,silero,"(0.6,)",99.68,84.71 +redux,silero,"(0.65,)",99.71,84.25 +redux,silero,"(0.7,)",99.74,83.78 +redux,silero,"(0.75,)",99.76,83.28 +redux,silero,"(0.8,)",99.8,82.68 +redux,silero,"(0.85,)",99.83,81.97 +redux,silero,"(0.9,)",99.86,80.88 +redux,head,"(0.1,)",89.05,97.92 +redux,head,"(0.15,)",90.17,97.26 +redux,head,"(0.2,)",91.33,96.64 +redux,head,"(0.25,)",92.49,96.04 +redux,head,"(0.3,)",93.63,95.26 +redux,head,"(0.35,)",94.82,94.45 +redux,head,"(0.4,)",95.86,93.64 +redux,head,"(0.45,)",96.85,92.81 +redux,head,"(0.5,)",97.58,92.05 +redux,head,"(0.55,)",98.16,91.33 +redux,head,"(0.6,)",98.63,90.72 +redux,head,"(0.65,)",98.93,90.08 +redux,head,"(0.7,)",99.08,89.48 +redux,head,"(0.75,)",99.22,88.79 +redux,head,"(0.8,)",99.32,88.02 +redux,head,"(0.85,)",99.43,87.1 +redux,head,"(0.9,)",99.56,85.84 +redux,or,"(0.1, 0.1)",89.02,98.05 +redux,or,"(0.2, 0.1)",89.05,97.95 +redux,or,"(0.3, 0.1)",89.05,97.93 +redux,or,"(0.4, 0.1)",89.05,97.93 +redux,or,"(0.5, 0.1)",89.05,97.93 +redux,or,"(0.1, 0.2)",91.14,96.97 +redux,or,"(0.2, 0.2)",91.29,96.73 +redux,or,"(0.3, 0.2)",91.32,96.68 +redux,or,"(0.4, 0.2)",91.33,96.66 +redux,or,"(0.5, 0.2)",91.33,96.65 +redux,or,"(0.1, 0.3)",93.25,95.91 +redux,or,"(0.2, 0.3)",93.53,95.48 +redux,or,"(0.3, 0.3)",93.6,95.35 +redux,or,"(0.4, 0.3)",93.62,95.3 +redux,or,"(0.5, 0.3)",93.63,95.27 +redux,or,"(0.6, 0.3)",93.63,95.27 +redux,or,"(0.7, 0.3)",93.63,95.26 +redux,or,"(0.8, 0.3)",93.63,95.26 +redux,or,"(0.9, 0.3)",93.63,95.26 +redux,or,"(0.1, 0.4)",95.3,94.8 +redux,or,"(0.2, 0.4)",95.72,94.09 +redux,or,"(0.3, 0.4)",95.81,93.84 +redux,or,"(0.1, 0.5)",96.8,93.76 +redux,or,"(0.1, 0.6)",97.71,93.04 +redux,or,"(0.1, 0.7)",98.05,92.53 +redux,or,"(0.1, 0.8)",98.17,92.1 +redux,or,"(0.2, 0.6)",98.35,91.77 +redux,or,"(0.3, 0.6)",98.52,91.27 +redux,or,"(0.2, 0.7)",98.73,91.04 +redux,or,"(0.2, 0.8)",98.89,90.42 +redux,or,"(0.3, 0.7)",98.93,90.37 +redux,or,"(0.4, 0.7)",99.01,90.02 +redux,or,"(0.5, 0.7)",99.05,89.81 +redux,or,"(0.6, 0.7)",99.06,89.69 +redux,or,"(0.7, 0.7)",99.07,89.62 +redux,or,"(0.8, 0.7)",99.08,89.58 +redux,or,"(0.9, 0.7)",99.08,89.54 +redux,or,"(0.3, 0.8)",99.11,89.54 +redux,or,"(0.4, 0.8)",99.21,89.04 +redux,or,"(0.3, 0.9)",99.24,88.77 +redux,or,"(0.5, 0.8)",99.26,88.71 +redux,or,"(0.6, 0.8)",99.29,88.49 +redux,or,"(0.7, 0.8)",99.3,88.35 +redux,or,"(0.8, 0.8)",99.31,88.25 +redux,or,"(0.9, 0.8)",99.31,88.18 +redux,or,"(0.4, 0.9)",99.37,88.07 +redux,or,"(0.5, 0.9)",99.44,87.56 +redux,or,"(0.6, 0.9)",99.48,87.15 +redux,or,"(0.7, 0.9)",99.51,86.84 +redux,or,"(0.8, 0.9)",99.53,86.61 +redux,or,"(0.9, 0.9)",99.55,86.39 +redux,and,"(0.1, 0.1)",98.36,90.88 +redux,and,"(0.1, 0.2)",98.54,90.59 +redux,and,"(0.1, 0.3)",98.73,90.2 +redux,and,"(0.1, 0.4)",98.91,89.64 +redux,and,"(0.1, 0.5)",99.1,89.15 +redux,and,"(0.2, 0.2)",99.13,88.67 +redux,and,"(0.1, 0.6)",99.25,88.54 +redux,and,"(0.2, 0.5)",99.31,87.98 +redux,and,"(0.1, 0.7)",99.36,87.84 +redux,and,"(0.2, 0.6)",99.37,87.59 +redux,and,"(0.3, 0.3)",99.38,87.4 +redux,and,"(0.3, 0.4)",99.4,87.26 +redux,and,"(0.3, 0.5)",99.43,87.12 +redux,and,"(0.2, 0.7)",99.43,87.07 +redux,and,"(0.1, 0.8)",99.45,86.87 +redux,and,"(0.3, 0.6)",99.46,86.87 +redux,and,"(0.4, 0.1)",99.5,86.48 +redux,and,"(0.3, 0.7)",99.5,86.47 +redux,and,"(0.4, 0.2)",99.5,86.47 +redux,and,"(0.4, 0.3)",99.51,86.44 +redux,and,"(0.4, 0.4)",99.52,86.38 +redux,and,"(0.4, 0.5)",99.53,86.29 +redux,and,"(0.4, 0.6)",99.55,86.12 +redux,and,"(0.3, 0.8)",99.56,85.86 +redux,and,"(0.4, 0.7)",99.57,85.85 +redux,and,"(0.5, 0.1)",99.61,85.56 +redux,and,"(0.5, 0.2)",99.61,85.55 +redux,and,"(0.5, 0.3)",99.61,85.53 +redux,and,"(0.5, 0.4)",99.61,85.5 +redux,and,"(0.5, 0.5)",99.62,85.46 +redux,and,"(0.5, 0.6)",99.63,85.36 +redux,and,"(0.5, 0.7)",99.64,85.16 +redux,and,"(0.2, 0.9)",99.64,84.84 +redux,and,"(0.5, 0.8)",99.67,84.79 +redux,and,"(0.6, 0.1)",99.68,84.7 +redux,and,"(0.6, 0.2)",99.68,84.7 +redux,and,"(0.6, 0.3)",99.68,84.69 +redux,and,"(0.6, 0.4)",99.68,84.67 +redux,and,"(0.6, 0.5)",99.68,84.65 +redux,and,"(0.6, 0.6)",99.69,84.58 +redux,and,"(0.6, 0.7)",99.69,84.44 +redux,and,"(0.6, 0.8)",99.71,84.15 +redux,and,"(0.5, 0.9)",99.73,83.81 +redux,and,"(0.7, 0.1)",99.74,83.78 +redux,and,"(0.7, 0.2)",99.74,83.78 +redux,and,"(0.7, 0.3)",99.74,83.78 +redux,and,"(0.7, 0.4)",99.74,83.76 +redux,and,"(0.7, 0.5)",99.74,83.75 +redux,and,"(0.7, 0.6)",99.74,83.71 +redux,and,"(0.7, 0.7)",99.74,83.61 +redux,and,"(0.7, 0.8)",99.76,83.4 +redux,and,"(0.7, 0.9)",99.78,82.69 +redux,and,"(0.8, 0.1)",99.8,82.68 +redux,and,"(0.8, 0.2)",99.8,82.67 +redux,and,"(0.8, 0.3)",99.8,82.67 +redux,and,"(0.8, 0.4)",99.8,82.66 +redux,and,"(0.8, 0.5)",99.8,82.66 +redux,and,"(0.8, 0.6)",99.8,82.63 +redux,and,"(0.8, 0.7)",99.8,82.56 +redux,and,"(0.8, 0.8)",99.81,82.41 +redux,and,"(0.8, 0.9)",99.82,81.84 +redux,and,"(0.9, 0.1)",99.86,80.88 +redux,and,"(0.9, 0.3)",99.86,80.88 +redux,and,"(0.9, 0.4)",99.86,80.87 +redux,and,"(0.9, 0.5)",99.86,80.87 +redux,and,"(0.9, 0.7)",99.86,80.81 +redux,and,"(0.9, 0.8)",99.86,80.71 +redux,and,"(0.9, 0.9)",99.87,80.27 +redux,mean,"(0.25, 0.1)",89.55,97.7 +redux,mean,"(0.5, 0.1)",90.61,97.22 +redux,mean,"(0.25, 0.15)",91.05,96.88 +redux,mean,"(0.25, 0.2)",92.59,96.09 +redux,mean,"(0.5, 0.15)",92.83,96.06 +redux,mean,"(0.75, 0.1)",93.73,95.89 +redux,mean,"(0.25, 0.25)",94.15,95.09 +redux,mean,"(0.5, 0.2)",95.1,94.74 +redux,mean,"(0.25, 0.3)",95.62,94.06 +redux,mean,"(0.75, 0.15)",97.11,93.64 +redux,mean,"(0.25, 0.4)",97.77,92.07 +redux,mean,"(0.5, 0.3)",98.12,92.04 +redux,mean,"(0.75, 0.2)",98.46,91.79 +redux,mean,"(0.25, 0.45)",98.48,91.26 +redux,mean,"(0.5, 0.35)",98.75,91.05 +redux,mean,"(0.25, 0.5)",98.87,90.56 +redux,mean,"(0.75, 0.25)",98.94,90.3 +redux,mean,"(0.5, 0.4)",99.04,90.17 +redux,mean,"(0.25, 0.55)",99.05,89.88 +redux,mean,"(0.5, 0.45)",99.2,89.27 +redux,mean,"(0.25, 0.65)",99.32,88.36 +redux,mean,"(0.5, 0.5)",99.34,88.2 +redux,mean,"(0.75, 0.4)",99.38,87.57 +redux,mean,"(0.25, 0.7)",99.43,87.52 +redux,mean,"(0.5, 0.55)",99.44,87.26 +redux,mean,"(0.75, 0.45)",99.46,86.98 +redux,mean,"(0.5, 0.6)",99.52,86.56 +redux,mean,"(0.75, 0.5)",99.52,86.43 +redux,mean,"(0.25, 0.75)",99.55,86.39 +redux,mean,"(0.5, 0.65)",99.58,85.94 +redux,mean,"(0.75, 0.6)",99.63,85.4 +redux,mean,"(0.5, 0.7)",99.64,85.31 +redux,mean,"(0.75, 0.65)",99.67,84.92 +redux,mean,"(0.5, 0.75)",99.69,84.65 +redux,mean,"(0.75, 0.7)",99.7,84.38 +redux,mean,"(0.25, 0.85)",99.71,84.23 +redux,mean,"(0.5, 0.8)",99.74,83.88 +redux,mean,"(0.75, 0.75)",99.74,83.82 +redux,mean,"(0.75, 0.8)",99.77,83.18 +redux,mean,"(0.5, 0.85)",99.78,83.03 +redux,mean,"(0.25, 0.9)",99.78,82.8 +redux,mean,"(0.75, 0.85)",99.81,82.4 +redux,mean,"(0.5, 0.9)",99.83,81.87 +redux,mean,"(0.75, 0.9)",99.85,81.32 +redux,max,"(0.1,)",89.02,98.05 +redux,max,"(0.15,)",90.14,97.36 +redux,max,"(0.2,)",91.29,96.73 +redux,max,"(0.25,)",92.45,96.14 +redux,max,"(0.3,)",93.6,95.35 +redux,max,"(0.35,)",94.8,94.54 +redux,max,"(0.4,)",95.84,93.73 +redux,max,"(0.45,)",96.84,92.9 +redux,max,"(0.5,)",97.56,92.12 +redux,max,"(0.55,)",98.15,91.42 +redux,max,"(0.6,)",98.63,90.83 +redux,max,"(0.65,)",98.92,90.21 +redux,max,"(0.7,)",99.07,89.62 +redux,max,"(0.75,)",99.21,88.97 +redux,max,"(0.8,)",99.31,88.25 +redux,max,"(0.85,)",99.42,87.44 +redux,max,"(0.9,)",99.55,86.39 +redux,two,"(0.3, 0.3, 48.0, 32.0, 60.0)",97.42,94.27 +redux,two,"(0.3, 0.3, 48.0, 32.0, 30.0)",97.57,94.18 +redux,two,"(0.3, 0.3, 32.0, 32.0, 60.0)",97.61,94.16 +redux,two,"(0.5, 0.3, 48.0, 32.0, 60.0)",97.63,94.09 +redux,two,"(0.3, 0.3, 32.0, 32.0, 30.0)",97.74,94.05 +redux,two,"(0.3, 0.3, 48.0, 16.0, 60.0)",97.8,94.05 +redux,two,"(0.5, 0.3, 32.0, 32.0, 60.0)",97.82,93.97 +redux,two,"(0.3, 0.3, 32.0, 16.0, 60.0)",97.99,93.94 +redux,two,"(0.5, 0.3, 48.0, 16.0, 60.0)",98.01,93.77 +redux,two,"(0.3, 0.3, 32.0, 16.0, 30.0)",98.08,93.75 +redux,two,"(0.3, 0.3, 32.0, 16.0, 0.0)",98.08,93.7 +redux,two,"(0.5, 0.3, 32.0, 16.0, 60.0)",98.2,93.64 +redux,two,"(0.3, 0.3, 32.0, 8.0, 60.0)",98.23,93.58 +redux,two,"(0.5, 0.3, 32.0, 16.0, 30.0)",98.28,93.38 +redux,two,"(0.7, 0.3, 32.0, 16.0, 60.0)",98.33,93.34 +redux,two,"(0.5, 0.3, 16.0, 16.0, 60.0)",98.4,93.3 +redux,two,"(0.3, 0.3, 16.0, 8.0, 60.0)",98.43,93.27 +redux,two,"(0.5, 0.3, 32.0, 8.0, 60.0)",98.51,92.98 +redux,two,"(0.7, 0.3, 16.0, 16.0, 60.0)",98.52,92.92 +redux,two,"(0.5, 0.3, 8.0, 16.0, 60.0)",98.58,92.7 +redux,two,"(0.3, 0.3, 8.0, 8.0, 60.0)",98.61,92.69 +redux,two,"(0.5, 0.3, 16.0, 8.0, 60.0)",98.7,92.63 +redux,two,"(0.7, 0.3, 32.0, 8.0, 60.0)",98.71,92.37 +redux,two,"(0.3, 0.3, 16.0, 0.0, 60.0)",98.86,92.07 +redux,two,"(0.7, 0.3, 16.0, 8.0, 60.0)",98.89,91.96 +redux,two,"(0.5, 0.3, 32.0, 0.0, 60.0)",98.97,91.57 +redux,two,"(0.3, 0.3, 8.0, 0.0, 60.0)",99.04,91.5 +redux,two,"(0.5, 0.3, 16.0, 0.0, 60.0)",99.16,91.17 +redux,two,"(0.5, 0.3, 8.0, 0.0, 60.0)",99.33,90.5 +redux,two,"(0.7, 0.3, 8.0, 0.0, 60.0)",99.49,89.7 +redux,two,"(0.5, 0.3, 0.0, 0.0, 60.0)",99.57,89.55 +redux,two,"(0.7, 0.3, 0.0, 0.0, 60.0)",99.69,88.71 +redux,two,"(0.7, 0.5, 0.0, 0.0, 60.0)",99.75,87.2 +redux,lr6_oof,thr=0.10,85.56,99.76 +redux,lr6_oof,thr=0.15,86.68,99.48 +redux,lr6_oof,thr=0.20,87.82,99.16 +redux,lr6_oof,thr=0.25,89.23,98.72 +redux,lr6_oof,thr=0.30,90.63,98.19 +redux,lr6_oof,thr=0.35,92.19,97.65 +redux,lr6_oof,thr=0.40,93.66,97.07 +redux,lr6_oof,thr=0.45,95.0,96.42 +redux,lr6_oof,thr=0.50,96.14,95.72 +redux,lr6_oof,thr=0.55,97.03,95.01 +redux,lr6_oof,thr=0.60,97.71,94.28 +redux,lr6_oof,thr=0.65,98.17,93.52 +redux,lr6_oof,thr=0.70,98.55,92.7 +redux,lr6_oof,thr=0.75,98.84,91.75 +redux,lr6_oof,thr=0.80,99.09,90.69 +redux,lr6_oof,thr=0.85,99.35,89.39 +redux,lr6_oof,thr=0.90,99.58,87.68 +redux,hgb6_oof,thr=0.10,84.83,99.95 +redux,hgb6_oof,thr=0.15,85.95,99.79 +redux,hgb6_oof,thr=0.20,89.35,99.02 +redux,hgb6_oof,thr=0.25,92.08,98.18 +redux,hgb6_oof,thr=0.30,93.68,97.58 +redux,hgb6_oof,thr=0.35,95.01,96.94 +redux,hgb6_oof,thr=0.40,95.98,96.39 +redux,hgb6_oof,thr=0.45,96.56,95.96 +redux,hgb6_oof,thr=0.50,96.98,95.61 +redux,hgb6_oof,thr=0.55,97.34,95.22 +redux,hgb6_oof,thr=0.60,97.66,94.82 +redux,hgb6_oof,thr=0.65,97.92,94.41 +redux,hgb6_oof,thr=0.70,98.18,93.86 +redux,hgb6_oof,thr=0.75,98.49,93.2 +redux,hgb6_oof,thr=0.80,98.84,92.17 +redux,hgb6_oof,thr=0.85,99.17,90.9 +redux,hgb6_oof,thr=0.90,99.48,89.45 +redux,lr6nz_oof,thr=0.10,85.96,99.43 +redux,lr6nz_oof,thr=0.15,87.6,98.98 +redux,lr6nz_oof,thr=0.20,89.65,98.41 +redux,lr6nz_oof,thr=0.25,91.45,97.84 +redux,lr6nz_oof,thr=0.30,93.03,97.3 +redux,lr6nz_oof,thr=0.35,94.36,96.7 +redux,lr6nz_oof,thr=0.40,95.39,96.1 +redux,lr6nz_oof,thr=0.45,96.21,95.48 +redux,lr6nz_oof,thr=0.50,96.9,94.9 +redux,lr6nz_oof,thr=0.55,97.46,94.29 +redux,lr6nz_oof,thr=0.60,97.88,93.68 +redux,lr6nz_oof,thr=0.65,98.24,93.04 +redux,lr6nz_oof,thr=0.70,98.53,92.33 +redux,lr6nz_oof,thr=0.75,98.78,91.57 +redux,lr6nz_oof,thr=0.80,99.04,90.62 +redux,lr6nz_oof,thr=0.85,99.26,89.52 +redux,lr6nz_oof,thr=0.90,99.49,88.1 diff --git a/scripts/vad_bench/fusion/prep_libri.py b/scripts/vad_bench/fusion/prep_libri.py new file mode 100644 index 0000000..3dafd8f --- /dev/null +++ b/scripts/vad_bench/fusion/prep_libri.py @@ -0,0 +1,12 @@ +import io, numpy as np, soundfile as sf, os +os.environ["HF_HOME"]=os.path.abspath("hf") +from datasets import load_dataset, Audio +ds=load_dataset("openslr/librispeech_asr","clean",split="test",streaming=True).cast_column("audio",Audio(decode=False)) +os.makedirs("data",exist_ok=True) +U=[];spk=[] +for i,ex in enumerate(ds): + if i%13: continue + y,sr=sf.read(io.BytesIO(ex["audio"]["bytes"]),dtype="float32"); assert sr==16000 + U.append(y);spk.append(int(ex["speaker_id"])) +np.save("data/libri.npy",np.array(U,dtype=object),allow_pickle=True); np.save("data/spk.npy",np.array(spk)) +print(len(U),len(set(spk)),flush=True) diff --git a/scripts/vad_bench/fusion/report.py b/scripts/vad_bench/fusion/report.py new file mode 100644 index 0000000..a2b0eec --- /dev/null +++ b/scripts/vad_bench/fusion/report.py @@ -0,0 +1,109 @@ +from analyze import * +import rules +out=[] +KL=(("lr2","5a logistic reg, 2 probs"),("lr6","5b logistic reg, + prev/next frame"),("hgb6","5c gradient boosting, + prev/next frame"),("lr6nz","5d as 5b, trained also on noise-only clips"),("hgb6nz","5e as 5c, trained also on noise-only clips")) +KINDS=[k for k,_ in KL] +def P_(s=""): out.append(s) +def crossfit_fixed(getarr,thr_idx_of): + rows=np.zeros((N,5)) + for f in range(4): + ps,arr=getarr("f%d"%f); te=np.flatnonzero(fold==f); rows[te]=arr[thr_idx_of(ps)][te] + return rows +def shiftfit(getarr,crit): + return crossfit(getarr,crit,"shift") +sysdefs={} # name -> (rows, note) +def build(h): + S={} + S["Silero thr .5 (unified post)"]=(A_of("silero",(0.5,),h),"fixed") + S["Silero thr .5 (native defaults: min speech .25, pause .1, pad .03)"]=(CFG[("silero",(0.5,),"ultra",0.25)],"fixed") + S[f"{h} head thr .5"]=(A_of("head",(0.5,),h),"fixed") + S["Silero tuned thr (CV)"]=crossfit(static("silero",h),"f1")[0],"cv" + S[f"{h} head tuned thr (CV)"]=crossfit(static("head",h),"f1")[0],"cv" + dflt=dict(OR=("or",(0.5,0.5)),AND=("and",(0.5,0.5)),MEAN=("mean",(0.5,0.5)),MAX=("max",(0.5,)),MIN=("min",(0.5,)),TWO=("two",(0.5,0.5,16,16,30)),SWITCH=("switch",(0.5,0.5,12))) + names=dict(OR="1 OR of decisions",AND="2 AND of decisions",MEAN="3 mean of probs (w=.5)",MAX="4a max of probs",MIN="4b min of probs",TWO="6 two-stage (Silero + head extend/fill)",SWITCH="7 noise-switch (estimated SNR)") + for k,(r,p) in dflt.items(): S[f"{names[k]} @ defaults {p}"]=(A_of(r,p,h),"fixed") + for kind,nm in KL: + S[f"{nm} @ thr .5"]=(crossfit_fixed(lrget(h,kind),lambda ps:ps.index((0.5,))),"cv-model") + for k in ("OR","AND","MAX","MIN","TWO","SWITCH"): + r=dflt[k][0]; S[f"{names[k]} tuned (CV)"]=crossfit(static(r,h),"f1")[0],"cv" + # mean: separate weights are in the same candidate set + S[f"{names['MEAN']} tuned w,thr (CV)"]=crossfit(static("mean",h),"f1")[0],"cv" + for kind,nm in KL: + S[f"{nm} tuned thr (CV)"]=crossfit(lrget(h,kind),"f1")[0],"cv" + S["7o oracle noise-switch (true SNR) tuned (CV)"]=crossfit(static("oracle",h),"f1")[0],"cv" + return S +def table(S,title): + P_(f"### {title}"); P_() + P_("pooled over 342 speech clips; point [95% bootstrap CI over clips], percent. Per-level columns: F1 with CI (white+pink pooled).") + P_() + P_("| system | P | R | F1 | F1 clean | F1 20 dB | F1 10 dB | F1 5 dB | F1 0 dB |"); P_("|---|---|---|---|---|---|---|---|---|") + for n,(rows,_) in S.items(): + s=stats(rows); a=s["all"] + P_(f"| {n} | {fmt(a['P'],*a['Pci'])} | {fmt(a['R'],*a['Rci'])} | {fmt(a['F'],*a['Fci'])} | "+" | ".join(fmt(s[l]['F'],*s[l]['Fci']) for l,_ in LEVELS)+" |") + P_() +def table_pr_levels(S,title,names): + P_(f"### {title}"); P_(); P_("| system | "+" | ".join(f"P/R {l}" for l,_ in LEVELS)+" |"); P_("|---|"+"---|"*len(LEVELS)) + for n in names: + s=stats(S[n][0]); P_(f"| {n} | "+" | ".join(f"{100*s[l]['P']:.1f}/{100*s[l]['R']:.1f}" for l,_ in LEVELS)+" |") + P_() +def fa_table(S,names,title): + P_(f"### {title}"); P_(); P_("noise-only clips (12 x 30 s per level, white+pink pooled over the levels shown). False-alarm segments per hour (frame false-positive rate in brackets).") + P_(); P_("| system | "+" | ".join(l for l,_ in LEVELS[1:])+" | all |"); P_("|---|"+"---|"*5) + for n in names: + f=fa(S[n][0]); P_(f"| {n} | "+" | ".join(f"{f[l][0]:.0f} ({100*f[l][1]:.2f}%)" for l,_ in LEVELS[1:])+f" | {f['all'][0]:.0f} ({100*f['all'][1]:.2f}%) |") + P_() +ALL={} +for h in HEADS: + S=build(h); ALL[h]=S + P_(f"## Head = {h}"); P_() + table(S,f"Cross-validated results ({h} head + Silero; speaker-disjoint 4-fold; 'tuned' = parameters chosen on the 3 training folds, scored on the held-out fold)") + tuned=[n for n in S if "tuned" in n or "@" in n or "head thr" in n or "Silero thr" in n] + fa_table(S,[n for n in S],"False alarms on noise-only clips") + # best fusion vs best single + singles=["Silero tuned thr (CV)",f"{h} head tuned thr (CV)"] + fus=[n for n in S if n not in singles and "thr .5 (" not in n and "head thr" not in n and "native" not in n and "oracle" not in n] + bs=max(singles,key=lambda n:stats(S[n][0])["all"]["F"]); + P_(f"**Best single (CV-tuned):** {bs}."); P_() + P_("| fusion (CV) | dF1 vs best single, pooled | dF1 5 dB | dF1 0 dB | dR pooled | dP pooled |"); P_("|---|---|---|---|---|---|") + b0=stats(S[bs][0]) + for n in fus: + s=stats(S[n][0]); d=s["all"]["F_b"]-b0["all"]["F_b"] + # recall / precision diff, paired + idx,bi=bidx("all",np.flatnonzero(sp)); Pb,Rb,_=prf_b(S[n][0],idx,bi); Pb0,Rb0,_=prf_b(S[bs][0],idx,bi) + l5=s["5 dB"]["F_b"]-b0["5 dB"]["F_b"]; l0=s["0 dB"]["F_b"]-b0["0 dB"]["F_b"] + f=lambda x,y: f"{100*x:+.2f} [{100*y[0]:+.2f},{100*y[1]:+.2f}]" + P_(f"| {n} | {f(s['all']['F']-b0['all']['F'],ci(d))} | {f(s['5 dB']['F']-b0['5 dB']['F'],ci(l5))} | {f(s['0 dB']['F']-b0['0 dB']['F'],ci(l0))} | {f(s['all']['R']-b0['all']['R'],ci(Rb-Rb0))} | {f(s['all']['P']-b0['all']['P'],ci(Pb-Pb0))} |") + P_() + # shift generalisation + P_(f"### Noise-shift split ({h}): parameters chosen on clean/20/10 dB clips only, scored on 5 and 0 dB clips"); P_() + P_("| system | P | R | F1 (5+0 dB clips) |"); P_("|---|---|---|---|") + def shiftrow(nm,getarr): + rows,ch=shiftfit(getarr,"f1"); idx=np.flatnonzero(sp&(snr<10)); ii=rng.integers(0,len(idx),(B,len(idx))); Pb,Rb,Fb=prf_b(rows,idx,ii); p,r,f=pooled(rows,idx) + P_(f"| {nm} {ch['shift']} | {fmt(p,*ci(Pb))} | {fmt(r,*ci(Rb))} | {fmt(f,*ci(Fb))} |") + for r_,nm in (("silero","Silero"),("head",h+" head"),("or","OR"),("and","AND"),("mean","mean"),("max","max"),("min","min"),("two","two-stage"),("switch","noise-switch")): + shiftrow(nm,static(r_,h)) + for kind in KINDS: shiftrow(kind,lrget(h,kind)) + P_() + # precision-constrained + P_(f"### Operating points with precision >= 99% ({h}): CV-selected maximum recall") + P_(); P_("| system | P | R | F1 | chosen params per fold |"); P_("|---|---|---|---|---|") + for r_,nm in (("silero","Silero"),("head",h+" head"),("or","OR"),("and","AND"),("mean","mean"),("max","max"),("min","min"),("two","two-stage"),("switch","noise-switch")): + rows,ch=crossfit(static(r_,h),0.99); s=stats(rows)["all"]; P_(f"| {nm} | {fmt(s['P'],*s['Pci'])} | {fmt(s['R'],*s['Rci'])} | {fmt(s['F'],*s['Fci'])} | {list(ch.values())} |") + for kind in KINDS: + rows,ch=crossfit(lrget(h,kind),0.99); s=stats(rows)["all"]; P_(f"| {kind} | {fmt(s['P'],*s['Pci'])} | {fmt(s['R'],*s['Rci'])} | {fmt(s['F'],*s['Fci'])} | thr {[c[0] for c in ch.values()]} |") + P_() + # in-sample frontier (descriptive) + P_(f"### In-sample best operating points ({h}), descriptive only (selected on all clips, optimistic)"); P_() + P_("| rule | best F1 params | P | R | F1 | max R at P>=99% params | P | R |"); P_("|---|---|---|---|---|---|---|---|") + allidx=np.flatnonzero(sp) + for r_ in("silero","head","or","and","mean","max","min","two","switch","oracle"): + ps,arr=cand(r_,h); i=score(arr,allidx,"f1"); j=score(arr,allidx,0.99) + a=prf(arr[i][allidx,:4].sum(0)); b=prf(arr[j][allidx,:4].sum(0)) + P_(f"| {r_} | {ps[i]} | {100*a[0]:.1f} | {100*a[1]:.1f} | {100*a[2]:.1f} | {ps[j]} | {100*b[0]:.1f} | {100*b[1]:.1f} |") + P_() + # LR coefs + P_(f"LR coefficients ({h}, fit on folds != 0; features lg(silero), lg(head)[, lg(silero prev/next), lg(head prev/next)]):") + for kind in("lr2","lr6","lr6nz"): P_(f"- {kind}: {LR[(h,kind,'f0')][1]}") + P_() +open("results.md","w").write("\n".join(out)); pickle.dump({h:{n:v[0] for n,v in S.items()} for h,S in ALL.items()},open("rows.pkl","wb")) +print("\n".join(out)[:200]) diff --git a/scripts/vad_bench/fusion/results.md b/scripts/vad_bench/fusion/results.md new file mode 100644 index 0000000..96d8c10 --- /dev/null +++ b/scripts/vad_bench/fusion/results.md @@ -0,0 +1,325 @@ +## Head = ultra + +### Cross-validated results (ultra head + Silero; speaker-disjoint 4-fold; 'tuned' = parameters chosen on the 3 training folds, scored on the held-out fold) + +pooled over 342 speech clips; point [95% bootstrap CI over clips], percent. Per-level columns: F1 with CI (white+pink pooled). + +| system | P | R | F1 | F1 clean | F1 20 dB | F1 10 dB | F1 5 dB | F1 0 dB | +|---|---|---|---|---|---|---|---|---| +| Silero thr .5 (unified post) | 99.6 [99.6,99.6] | 85.6 [84.9,86.2] | 92.1 [91.7,92.4] | 93.5 [92.5,94.4] | 92.8 [92.0,93.5] | 92.4 [91.6,93.1] | 91.9 [91.2,92.7] | 90.4 [89.4,91.2] | +| Silero thr .5 (native defaults: min speech .25, pause .1, pad .03) | 99.4 [99.3,99.4] | 86.7 [86.1,87.3] | 92.6 [92.3,92.9] | 93.9 [92.9,94.7] | 93.3 [92.6,93.9] | 93.0 [92.3,93.6] | 92.5 [91.9,93.2] | 90.9 [90.1,91.8] | +| ultra head thr .5 | 98.6 [98.3,98.8] | 89.4 [88.8,90.0] | 93.8 [93.5,94.1] | 95.0 [94.2,95.7] | 94.2 [93.6,94.8] | 93.6 [93.0,94.2] | 93.3 [92.6,94.0] | 93.3 [92.7,93.9] | +| Silero tuned thr (CV) | 98.3 [98.1,98.4] | 91.1 [90.5,91.6] | 94.5 [94.2,94.8] | 94.9 [93.9,95.6] | 94.4 [93.8,95.0] | 94.6 [94.0,95.2] | 94.6 [94.0,95.2] | 94.3 [93.7,94.8] | +| ultra head tuned thr (CV) | 95.8 [95.2,96.3] | 93.0 [92.4,93.5] | 94.4 [94.1,94.7] | 95.6 [94.8,96.3] | 95.1 [94.6,95.6] | 94.9 [94.3,95.4] | 94.6 [94.0,95.1] | 92.5 [91.9,93.1] | +| 1 OR of decisions @ defaults (0.5, 0.5) | 98.5 [98.3,98.7] | 90.5 [89.9,91.0] | 94.3 [94.1,94.6] | 95.1 [94.3,95.8] | 94.6 [94.0,95.2] | 94.3 [93.7,94.9] | 94.2 [93.5,94.7] | 93.9 [93.3,94.5] | +| 2 AND of decisions @ defaults (0.5, 0.5) | 99.7 [99.6,99.7] | 84.4 [83.7,85.1] | 91.4 [91.0,91.8] | 93.3 [92.2,94.2] | 92.3 [91.6,93.0] | 91.6 [90.8,92.4] | 91.1 [90.2,91.8] | 89.7 [88.7,90.6] | +| 3 mean of probs (w=.5) @ defaults (0.5, 0.5) | 99.4 [99.4,99.5] | 87.1 [86.5,87.7] | 92.9 [92.5,93.2] | 94.2 [93.2,95.0] | 93.5 [92.8,94.1] | 93.0 [92.3,93.7] | 92.6 [91.9,93.3] | 91.7 [90.9,92.5] | +| 4a max of probs @ defaults (0.5,) | 98.5 [98.3,98.7] | 90.5 [89.9,91.0] | 94.3 [94.1,94.6] | 95.1 [94.3,95.8] | 94.6 [94.0,95.2] | 94.3 [93.7,94.9] | 94.2 [93.5,94.7] | 93.9 [93.3,94.5] | +| 4b min of probs @ defaults (0.5,) | 99.7 [99.6,99.7] | 84.4 [83.7,85.1] | 91.4 [91.0,91.8] | 93.3 [92.2,94.2] | 92.3 [91.6,93.0] | 91.6 [90.8,92.4] | 91.1 [90.2,91.8] | 89.7 [88.7,90.6] | +| 6 two-stage (Silero + head extend/fill) @ defaults (0.5, 0.5, 16, 16, 30) | 99.0 [99.0,99.1] | 90.0 [89.4,90.5] | 94.3 [94.0,94.6] | 95.1 [94.3,95.8] | 94.6 [94.0,95.1] | 94.3 [93.7,94.8] | 94.1 [93.4,94.6] | 93.8 [93.0,94.5] | +| 7 noise-switch (estimated SNR) @ defaults (0.5, 0.5, 12) | 98.9 [98.7,99.1] | 87.5 [86.8,88.1] | 92.8 [92.5,93.2] | 93.5 [92.5,94.4] | 92.8 [92.0,93.5] | 92.4 [91.6,93.1] | 92.6 [91.9,93.3] | 93.3 [92.7,93.9] | +| 5a logistic reg, 2 probs @ thr .5 | 95.7 [95.2,96.1] | 94.6 [94.1,95.1] | 95.1 [94.9,95.4] | 95.8 [95.1,96.5] | 95.5 [95.0,96.1] | 95.6 [95.1,96.1] | 95.2 [94.7,95.7] | 93.9 [93.4,94.4] | +| 5b logistic reg, + prev/next frame @ thr .5 | 95.9 [95.5,96.3] | 95.4 [94.9,95.8] | 95.6 [95.4,95.9] | 96.1 [95.3,96.7] | 96.0 [95.5,96.5] | 96.1 [95.7,96.5] | 95.8 [95.4,96.2] | 94.4 [93.9,94.9] | +| 5c gradient boosting, + prev/next frame @ thr .5 | 96.7 [96.5,96.9] | 95.8 [95.4,96.2] | 96.3 [96.1,96.4] | 97.0 [96.5,97.4] | 96.6 [96.2,96.9] | 96.4 [96.0,96.7] | 96.2 [95.8,96.5] | 95.6 [95.2,96.0] | +| 5d as 5b, trained also on noise-only clips @ thr .5 | 96.6 [96.4,96.9] | 94.7 [94.2,95.1] | 95.6 [95.4,95.8] | 96.0 [95.2,96.7] | 95.9 [95.3,96.4] | 95.8 [95.3,96.2] | 95.6 [95.2,96.0] | 95.0 [94.5,95.5] | +| 5e as 5c, trained also on noise-only clips @ thr .5 | 96.8 [96.6,97.0] | 95.6 [95.1,95.9] | 96.2 [96.0,96.4] | 96.9 [96.4,97.3] | 96.5 [96.1,96.9] | 96.3 [95.9,96.7] | 96.1 [95.7,96.5] | 95.4 [94.9,95.9] | +| 1 OR of decisions tuned (CV) | 97.6 [97.3,97.8] | 93.1 [92.6,93.5] | 95.3 [95.0,95.5] | 95.6 [94.7,96.2] | 95.1 [94.6,95.7] | 95.4 [94.8,95.9] | 95.4 [94.9,95.9] | 95.0 [94.6,95.5] | +| 2 AND of decisions tuned (CV) | 98.5 [98.4,98.7] | 90.5 [89.9,91.0] | 94.3 [94.1,94.6] | 94.7 [93.8,95.5] | 94.2 [93.6,94.8] | 94.3 [93.6,94.8] | 94.4 [93.8,95.0] | 94.3 [93.7,94.8] | +| 4a max of probs tuned (CV) | 96.1 [95.6,96.7] | 93.3 [92.8,93.8] | 94.7 [94.4,95.0] | 95.7 [94.9,96.3] | 95.3 [94.8,95.8] | 95.2 [94.6,95.7] | 94.9 [94.4,95.4] | 93.0 [92.4,93.6] | +| 4b min of probs tuned (CV) | 98.5 [98.4,98.7] | 90.5 [89.9,91.0] | 94.3 [94.1,94.6] | 94.7 [93.8,95.5] | 94.2 [93.6,94.8] | 94.3 [93.6,94.8] | 94.4 [93.8,95.0] | 94.3 [93.7,94.8] | +| 6 two-stage (Silero + head extend/fill) tuned (CV) | 98.5 [98.3,98.6] | 92.3 [91.8,92.8] | 95.3 [95.0,95.5] | 95.6 [94.8,96.3] | 95.2 [94.7,95.7] | 95.1 [94.5,95.6] | 95.1 [94.6,95.6] | 95.6 [95.0,96.1] | +| 7 noise-switch (estimated SNR) tuned (CV) | 96.7 [96.2,97.2] | 92.1 [91.6,92.6] | 94.4 [94.1,94.6] | 94.8 [93.9,95.5] | 95.0 [94.4,95.5] | 94.6 [94.0,95.2] | 94.5 [94.0,95.1] | 93.1 [92.5,93.7] | +| 3 mean of probs (w=.5) tuned w,thr (CV) | 96.6 [96.1,97.0] | 93.7 [93.2,94.2] | 95.1 [94.8,95.4] | 95.7 [94.9,96.4] | 95.4 [94.8,95.9] | 95.4 [94.8,95.9] | 95.4 [94.8,95.8] | 94.1 [93.5,94.6] | +| 5a logistic reg, 2 probs tuned thr (CV) | 96.7 [96.4,97.0] | 93.7 [93.2,94.2] | 95.2 [94.9,95.4] | 95.7 [94.8,96.3] | 95.4 [94.8,95.9] | 95.4 [94.8,95.9] | 95.4 [94.9,95.9] | 94.4 [93.9,94.9] | +| 5b logistic reg, + prev/next frame tuned thr (CV) | 96.7 [96.4,97.1] | 94.7 [94.2,95.1] | 95.7 [95.5,95.9] | 96.0 [95.2,96.6] | 95.8 [95.3,96.3] | 96.0 [95.5,96.4] | 96.0 [95.5,96.4] | 94.9 [94.4,95.4] | +| 5c gradient boosting, + prev/next frame tuned thr (CV) | 96.7 [96.5,96.9] | 95.8 [95.4,96.2] | 96.3 [96.1,96.4] | 97.0 [96.5,97.4] | 96.6 [96.2,96.9] | 96.4 [96.0,96.7] | 96.2 [95.8,96.5] | 95.6 [95.2,96.0] | +| 5d as 5b, trained also on noise-only clips tuned thr (CV) | 96.8 [96.6,97.1] | 94.6 [94.1,95.0] | 95.7 [95.5,95.9] | 96.1 [95.3,96.7] | 95.9 [95.3,96.4] | 95.9 [95.4,96.3] | 95.7 [95.3,96.1] | 95.1 [94.6,95.5] | +| 5e as 5c, trained also on noise-only clips tuned thr (CV) | 96.7 [96.5,96.9] | 95.7 [95.3,96.1] | 96.2 [96.0,96.4] | 97.0 [96.5,97.3] | 96.5 [96.2,96.9] | 96.3 [95.9,96.7] | 96.2 [95.8,96.5] | 95.4 [94.9,95.9] | +| 7o oracle noise-switch (true SNR) tuned (CV) | 96.9 [96.4,97.4] | 91.1 [90.6,91.7] | 93.9 [93.6,94.2] | 94.0 [93.0,94.9] | 93.4 [92.7,94.0] | 94.6 [94.0,95.2] | 94.5 [94.0,95.1] | 93.1 [92.5,93.7] | + +### False alarms on noise-only clips + +noise-only clips (12 x 30 s per level, white+pink pooled over the levels shown). False-alarm segments per hour (frame false-positive rate in brackets). + +| system | 20 dB | 10 dB | 5 dB | 0 dB | all | +|---|---|---|---|---|---| +| Silero thr .5 (unified post) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | +| Silero thr .5 (native defaults: min speech .25, pause .1, pad .03) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | +| ultra head thr .5 | 130 (99.33%) | 120 (99.56%) | 135 (99.32%) | 120 (99.53%) | 126 (99.44%) | +| Silero tuned thr (CV) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | +| ultra head tuned thr (CV) | 120 (99.73%) | 120 (99.81%) | 120 (99.81%) | 120 (99.78%) | 120 (99.78%) | +| 1 OR of decisions @ defaults (0.5, 0.5) | 130 (99.33%) | 120 (99.56%) | 135 (99.32%) | 120 (99.53%) | 126 (99.44%) | +| 2 AND of decisions @ defaults (0.5, 0.5) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | +| 3 mean of probs (w=.5) @ defaults (0.5, 0.5) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | +| 4a max of probs @ defaults (0.5,) | 130 (99.33%) | 120 (99.56%) | 135 (99.32%) | 120 (99.53%) | 126 (99.44%) | +| 4b min of probs @ defaults (0.5,) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | +| 6 two-stage (Silero + head extend/fill) @ defaults (0.5, 0.5, 16, 16, 30) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | +| 7 noise-switch (estimated SNR) @ defaults (0.5, 0.5, 12) | 130 (99.33%) | 120 (99.56%) | 135 (99.32%) | 120 (99.53%) | 126 (99.44%) | +| 5a logistic reg, 2 probs @ thr .5 | 1000 (18.21%) | 1265 (35.14%) | 720 (27.86%) | 465 (17.62%) | 862 (24.71%) | +| 5b logistic reg, + prev/next frame @ thr .5 | 800 (13.37%) | 1075 (27.54%) | 700 (23.74%) | 365 (15.73%) | 735 (20.09%) | +| 5c gradient boosting, + prev/next frame @ thr .5 | 750 (93.20%) | 535 (95.29%) | 450 (96.32%) | 475 (95.98%) | 552 (95.19%) | +| 5d as 5b, trained also on noise-only clips @ thr .5 | 5 (0.01%) | 5 (0.03%) | 15 (0.13%) | 15 (0.09%) | 10 (0.06%) | +| 5e as 5c, trained also on noise-only clips @ thr .5 | 55 (0.27%) | 45 (0.16%) | 60 (0.38%) | 45 (0.20%) | 51 (0.25%) | +| 1 OR of decisions tuned (CV) | 130 (99.33%) | 120 (99.56%) | 135 (99.32%) | 120 (99.53%) | 126 (99.44%) | +| 2 AND of decisions tuned (CV) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | +| 4a max of probs tuned (CV) | 120 (99.70%) | 120 (99.79%) | 120 (99.79%) | 120 (99.76%) | 120 (99.76%) | +| 4b min of probs tuned (CV) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | +| 6 two-stage (Silero + head extend/fill) tuned (CV) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | +| 7 noise-switch (estimated SNR) tuned (CV) | 120 (99.70%) | 120 (99.73%) | 120 (99.78%) | 120 (99.74%) | 120 (99.74%) | +| 3 mean of probs (w=.5) tuned w,thr (CV) | 145 (99.36%) | 140 (99.51%) | 145 (99.46%) | 125 (99.60%) | 139 (99.48%) | +| 5a logistic reg, 2 probs tuned thr (CV) | 755 (9.41%) | 1235 (22.73%) | 840 (19.75%) | 415 (13.41%) | 811 (16.32%) | +| 5b logistic reg, + prev/next frame tuned thr (CV) | 520 (7.06%) | 970 (18.34%) | 655 (16.74%) | 375 (11.34%) | 630 (13.37%) | +| 5c gradient boosting, + prev/next frame tuned thr (CV) | 750 (93.20%) | 535 (95.29%) | 450 (96.32%) | 475 (95.98%) | 552 (95.19%) | +| 5d as 5b, trained also on noise-only clips tuned thr (CV) | 5 (0.01%) | 5 (0.02%) | 10 (0.09%) | 10 (0.05%) | 8 (0.04%) | +| 5e as 5c, trained also on noise-only clips tuned thr (CV) | 55 (0.27%) | 55 (0.21%) | 65 (0.41%) | 50 (0.25%) | 56 (0.28%) | +| 7o oracle noise-switch (true SNR) tuned (CV) | 0 (0.00%) | 120 (99.73%) | 120 (99.78%) | 120 (99.74%) | 90 (74.81%) | + +**Best single (CV-tuned):** Silero tuned thr (CV). + +| fusion (CV) | dF1 vs best single, pooled | dF1 5 dB | dF1 0 dB | dR pooled | dP pooled | +|---|---|---|---|---|---| +| 1 OR of decisions @ defaults (0.5, 0.5) | -0.20 [-0.33,-0.06] | -0.47 [-0.71,-0.23] | -0.39 [-0.82,+0.06] | -0.57 [-0.83,-0.31] | +0.25 [+0.04,+0.44] | +| 2 AND of decisions @ defaults (0.5, 0.5) | -3.12 [-3.34,-2.90] | -3.57 [-4.03,-3.14] | -4.61 [-5.19,-4.05] | -6.62 [-6.99,-6.25] | +1.37 [+1.25,+1.50] | +| 3 mean of probs (w=.5) @ defaults (0.5, 0.5) | -1.66 [-1.82,-1.50] | -2.01 [-2.31,-1.71] | -2.59 [-3.01,-2.16] | -3.93 [-4.20,-3.67] | +1.16 [+1.04,+1.28] | +| 4a max of probs @ defaults (0.5,) | -0.20 [-0.33,-0.06] | -0.47 [-0.71,-0.23] | -0.39 [-0.82,+0.06] | -0.57 [-0.83,-0.31] | +0.25 [+0.04,+0.44] | +| 4b min of probs @ defaults (0.5,) | -3.12 [-3.34,-2.90] | -3.57 [-4.03,-3.14] | -4.61 [-5.19,-4.05] | -6.62 [-6.99,-6.25] | +1.37 [+1.25,+1.50] | +| 6 two-stage (Silero + head extend/fill) @ defaults (0.5, 0.5, 16, 16, 30) | -0.25 [-0.37,-0.14] | -0.56 [-0.80,-0.32] | -0.51 [-0.87,-0.17] | -1.09 [-1.30,-0.87] | +0.74 [+0.63,+0.86] | +| 7 noise-switch (estimated SNR) @ defaults (0.5, 0.5, 12) | -1.69 [-1.86,-1.51] | -2.02 [-2.39,-1.63] | -0.99 [-1.51,-0.45] | -3.59 [-3.92,-3.26] | +0.64 [+0.43,+0.84] | +| 5a logistic reg, 2 probs @ thr .5 | +0.61 [+0.42,+0.79] | +0.59 [+0.21,+0.96] | -0.36 [-0.96,+0.26] | +3.58 [+3.31,+3.86] | -2.63 [-2.94,-2.31] | +| 5b logistic reg, + prev/next frame @ thr .5 | +1.11 [+0.92,+1.29] | +1.20 [+0.83,+1.56] | +0.15 [-0.41,+0.74] | +4.30 [+4.05,+4.56] | -2.36 [-2.64,-2.09] | +| 5c gradient boosting, + prev/next frame @ thr .5 | +1.74 [+1.59,+1.90] | +1.54 [+1.27,+1.82] | +1.34 [+1.00,+1.68] | +4.77 [+4.54,+5.02] | -1.57 [-1.71,-1.44] | +| 5d as 5b, trained also on noise-only clips @ thr .5 | +1.09 [+0.96,+1.23] | +0.99 [+0.68,+1.31] | +0.72 [+0.37,+1.09] | +3.61 [+3.42,+3.80] | -1.68 [-1.82,-1.55] | +| 5e as 5c, trained also on noise-only clips @ thr .5 | +1.64 [+1.51,+1.79] | +1.50 [+1.24,+1.77] | +1.12 [+0.86,+1.38] | +4.51 [+4.28,+4.75] | -1.50 [-1.62,-1.38] | +| 1 OR of decisions tuned (CV) | +0.74 [+0.65,+0.84] | +0.81 [+0.69,+0.93] | +0.74 [+0.37,+1.13] | +2.02 [+1.85,+2.22] | -0.70 [-0.89,-0.54] | +| 2 AND of decisions tuned (CV) | -0.19 [-0.23,-0.15] | -0.21 [-0.31,-0.12] | -0.01 [-0.05,+0.04] | -0.55 [-0.62,-0.47] | +0.23 [+0.19,+0.28] | +| 4a max of probs tuned (CV) | +0.16 [-0.02,+0.32] | +0.31 [+0.02,+0.59] | -1.28 [-1.81,-0.77] | +2.24 [+1.93,+2.58] | -2.16 [-2.63,-1.68] | +| 4b min of probs tuned (CV) | -0.19 [-0.23,-0.15] | -0.21 [-0.31,-0.12] | -0.01 [-0.05,+0.04] | -0.55 [-0.62,-0.47] | +0.23 [+0.19,+0.28] | +| 6 two-stage (Silero + head extend/fill) tuned (CV) | +0.75 [+0.65,+0.86] | +0.52 [+0.34,+0.71] | +1.26 [+0.95,+1.58] | +1.26 [+1.06,+1.46] | +0.17 [+0.08,+0.28] | +| 7 noise-switch (estimated SNR) tuned (CV) | -0.18 [-0.35,-0.02] | -0.08 [-0.38,+0.22] | -1.17 [-1.69,-0.68] | +1.06 [+0.70,+1.43] | -1.58 [-2.03,-1.13] | +| 3 mean of probs (w=.5) tuned w,thr (CV) | +0.59 [+0.43,+0.72] | +0.75 [+0.50,+1.00] | -0.24 [-0.73,+0.23] | +2.68 [+2.40,+2.96] | -1.74 [-2.10,-1.40] | +| 5a logistic reg, 2 probs tuned thr (CV) | +0.66 [+0.52,+0.79] | +0.78 [+0.56,+1.00] | +0.11 [-0.37,+0.59] | +2.68 [+2.45,+2.91] | -1.59 [-1.83,-1.35] | +| 5b logistic reg, + prev/next frame tuned thr (CV) | +1.17 [+1.02,+1.31] | +1.33 [+1.07,+1.61] | +0.60 [+0.13,+1.10] | +3.63 [+3.41,+3.86] | -1.54 [-1.76,-1.32] | +| 5c gradient boosting, + prev/next frame tuned thr (CV) | +1.74 [+1.59,+1.90] | +1.54 [+1.27,+1.82] | +1.34 [+1.00,+1.68] | +4.77 [+4.54,+5.02] | -1.57 [-1.71,-1.44] | +| 5d as 5b, trained also on noise-only clips tuned thr (CV) | +1.16 [+1.03,+1.28] | +1.08 [+0.77,+1.39] | +0.80 [+0.46,+1.15] | +3.52 [+3.33,+3.71] | -1.45 [-1.58,-1.34] | +| 5e as 5c, trained also on noise-only clips tuned thr (CV) | +1.68 [+1.53,+1.83] | +1.54 [+1.27,+1.84] | +1.13 [+0.84,+1.42] | +4.67 [+4.43,+4.90] | -1.59 [-1.70,-1.47] | + +### Noise-shift split (ultra): parameters chosen on clean/20/10 dB clips only, scored on 5 and 0 dB clips + +| system | P | R | F1 (5+0 dB clips) | +|---|---|---|---| +| Silero (0.1,) | 97.6 [97.4,97.9] | 91.5 [90.6,92.3] | 94.5 [94.0,94.9] | +| ultra head (0.1,) | 86.5 [85.5,87.3] | 98.9 [98.5,99.2] | 92.2 [91.8,92.7] | +| OR (0.1, 0.1) | 86.3 [85.4,87.2] | 99.2 [98.9,99.4] | 92.3 [91.8,92.8] | +| AND (0.1, 0.1) | 97.9 [97.6,98.1] | 91.1 [90.2,91.9] | 94.3 [93.9,94.8] | +| mean (0.5, 0.1) | 89.3 [88.3,90.2] | 97.7 [97.2,98.2] | 93.3 [92.8,93.8] | +| max (0.1,) | 86.3 [85.4,87.2] | 99.2 [98.9,99.4] | 92.3 [91.8,92.8] | +| min (0.1,) | 97.9 [97.6,98.1] | 91.1 [90.2,91.9] | 94.3 [93.9,94.8] | +| two-stage (0.3, 0.3, 48.0, 16.0, 60.0) | 98.0 [97.7,98.2] | 92.8 [91.9,93.5] | 95.3 [94.9,95.7] | +| noise-switch (0.3, 0.3, 40.0) | 94.2 [93.2,95.1] | 93.4 [92.6,94.2] | 93.8 [93.4,94.2] | +| lr2 (0.5,) | 89.8 [89.0,90.5] | 97.8 [97.5,98.2] | 93.6 [93.2,94.0] | +| lr6 (0.45,) | 88.7 [87.8,89.5] | 98.8 [98.5,99.0] | 93.4 [93.0,93.9] | +| hgb6 (0.45,) | 91.1 [90.2,91.9] | 98.1 [97.7,98.4] | 94.4 [94.0,94.9] | +| lr6nz (0.5,) | 95.2 [94.7,95.5] | 95.5 [95.0,96.1] | 95.4 [95.0,95.6] | +| hgb6nz (0.45,) | 92.9 [92.3,93.4] | 96.9 [96.4,97.3] | 94.8 [94.4,95.2] | + +### Operating points with precision >= 99% (ultra): CV-selected maximum recall + +| system | P | R | F1 | chosen params per fold | +|---|---|---|---|---| +| Silero | 99.1 [99.0,99.2] | 88.6 [88.0,89.2] | 93.6 [93.2,93.9] | [(0.2,), (0.2,), (0.2,), (0.25,)] | +| ultra head | 99.0 [98.8,99.1] | 87.4 [86.8,88.1] | 92.9 [92.5,93.2] | [(0.55,), (0.6,), (0.65,), (0.65,)] | +| OR | 98.9 [98.7,99.0] | 89.7 [89.1,90.2] | 94.0 [93.7,94.3] | [(0.3, 0.6), (0.2, 0.8), (0.4, 0.7), (0.3, 0.8)] | +| AND | 99.1 [99.0,99.1] | 88.9 [88.3,89.4] | 93.7 [93.4,94.0] | [(0.1, 0.3), (0.1, 0.3), (0.2, 0.1), (0.2, 0.1)] | +| mean | 99.0 [98.9,99.1] | 89.5 [88.9,90.0] | 94.0 [93.7,94.3] | [(0.5, 0.35), (0.5, 0.35), (0.5, 0.4), (0.5, 0.4)] | +| max | 99.0 [98.8,99.1] | 88.8 [88.2,89.4] | 93.6 [93.3,93.9] | [(0.55,), (0.65,), (0.65,), (0.7,)] | +| min | 99.1 [99.0,99.2] | 88.7 [88.0,89.2] | 93.6 [93.3,93.9] | [(0.15,), (0.15,), (0.2,), (0.2,)] | +| two-stage | 99.0 [98.9,99.1] | 90.6 [90.0,91.1] | 94.6 [94.3,94.8] | [(0.3, 0.3, 16.0, 0.0, 60.0), (0.5, 0.3, 16.0, 8.0, 30.0), (0.3, 0.3, 8.0, 0.0, 60.0), (0.5, 0.3, 32.0, 0.0, 60.0)] | +| noise-switch | 99.0 [98.8,99.2] | 88.2 [87.5,88.7] | 93.3 [92.9,93.6] | [(0.3, 0.5, 16.0), (0.3, 0.5, 8.0), (0.3, 0.5, 8.0), (0.3, 0.5, 8.0)] | +| lr2 | 99.1 [99.1,99.2] | 89.1 [88.5,89.6] | 93.8 [93.5,94.1] | thr [0.8, 0.8, 0.8, 0.8] | +| lr6 | 99.1 [99.0,99.2] | 90.3 [89.7,90.8] | 94.5 [94.2,94.8] | thr [0.8, 0.8, 0.8, 0.8] | +| hgb6 | 99.1 [99.0,99.2] | 91.0 [90.5,91.5] | 94.9 [94.6,95.1] | thr [0.85, 0.85, 0.8, 0.8] | +| lr6nz | 99.1 [99.0,99.2] | 90.0 [89.4,90.6] | 94.3 [94.0,94.6] | thr [0.8, 0.8, 0.8, 0.8] | +| hgb6nz | 99.0 [98.9,99.1] | 91.3 [90.7,91.8] | 95.0 [94.7,95.2] | thr [0.8, 0.85, 0.8, 0.8] | + +### In-sample best operating points (ultra), descriptive only (selected on all clips, optimistic) + +| rule | best F1 params | P | R | F1 | max R at P>=99% params | P | R | +|---|---|---|---|---|---|---|---| +| silero | (0.1,) | 98.3 | 91.1 | 94.5 | (0.2,) | 99.1 | 88.8 | +| head | (0.25,) | 95.8 | 93.2 | 94.5 | (0.6,) | 99.0 | 87.8 | +| or | (0.1, 0.5) | 97.6 | 93.1 | 95.3 | (0.3, 0.7) | 99.0 | 89.7 | +| and | (0.1, 0.1) | 98.5 | 90.5 | 94.3 | (0.1, 0.3) | 99.0 | 88.9 | +| mean | (0.75, 0.1) | 96.1 | 94.3 | 95.2 | (0.75, 0.25) | 99.1 | 89.6 | +| max | (0.3,) | 96.6 | 93.0 | 94.7 | (0.65,) | 99.1 | 88.7 | +| min | (0.1,) | 98.5 | 90.5 | 94.3 | (0.2,) | 99.2 | 88.2 | +| two | (0.3, 0.3, 32.0, 16.0, 60.0) | 98.5 | 92.3 | 95.3 | (0.5, 0.3, 8.0, 8.0, 60.0) | 99.0 | 90.6 | +| switch | (0.3, 0.3, 40.0) | 96.7 | 92.1 | 94.4 | (0.3, 0.5, 8.0) | 99.2 | 87.8 | +| oracle | (0.3, 0.3, 10.0) | 96.9 | 91.1 | 93.9 | (0.3, 0.7, 5.0) | 99.4 | 86.3 | + +LR coefficients (ultra, fit on folds != 0; features lg(silero), lg(head)[, lg(silero prev/next), lg(head prev/next)]): +- lr2: ([0.494, 0.361], 2.1583826054377893) +- lr6: ([0.074, -0.41, 0.119, 0.368, 0.279, 0.533], 2.2670968869481705) +- lr6nz: ([-0.023, -0.518, 0.392, 0.604, 0.087, 0.484], 2.7718436816992393) + +## Head = redux + +### Cross-validated results (redux head + Silero; speaker-disjoint 4-fold; 'tuned' = parameters chosen on the 3 training folds, scored on the held-out fold) + +pooled over 342 speech clips; point [95% bootstrap CI over clips], percent. Per-level columns: F1 with CI (white+pink pooled). + +| system | P | R | F1 | F1 clean | F1 20 dB | F1 10 dB | F1 5 dB | F1 0 dB | +|---|---|---|---|---|---|---|---|---| +| Silero thr .5 (unified post) | 99.6 [99.6,99.6] | 85.6 [84.9,86.2] | 92.1 [91.7,92.4] | 93.5 [92.5,94.4] | 92.8 [92.0,93.5] | 92.4 [91.6,93.1] | 91.9 [91.2,92.7] | 90.4 [89.4,91.2] | +| Silero thr .5 (native defaults: min speech .25, pause .1, pad .03) | 99.4 [99.3,99.4] | 86.7 [86.1,87.3] | 92.6 [92.3,92.9] | 93.9 [92.9,94.7] | 93.3 [92.6,93.9] | 93.0 [92.3,93.6] | 92.5 [91.9,93.2] | 90.9 [90.1,91.8] | +| redux head thr .5 | 97.6 [97.3,97.8] | 92.0 [91.6,92.5] | 94.7 [94.5,95.0] | 95.4 [94.6,96.0] | 95.1 [94.5,95.6] | 94.8 [94.2,95.3] | 94.6 [94.1,95.1] | 94.1 [93.6,94.6] | +| Silero tuned thr (CV) | 98.3 [98.1,98.4] | 91.1 [90.5,91.6] | 94.5 [94.2,94.8] | 94.9 [93.9,95.6] | 94.4 [93.8,95.0] | 94.6 [94.0,95.2] | 94.6 [94.0,95.2] | 94.3 [93.7,94.8] | +| redux head tuned thr (CV) | 96.9 [96.5,97.2] | 92.8 [92.4,93.2] | 94.8 [94.5,95.0] | 95.5 [94.7,96.1] | 95.2 [94.6,95.7] | 94.9 [94.3,95.4] | 94.6 [94.0,95.1] | 94.2 [93.7,94.6] | +| 1 OR of decisions @ defaults (0.5, 0.5) | 97.6 [97.3,97.8] | 92.1 [91.7,92.5] | 94.8 [94.5,95.0] | 95.4 [94.6,96.1] | 95.1 [94.6,95.6] | 94.8 [94.2,95.3] | 94.7 [94.1,95.2] | 94.2 [93.7,94.6] | +| 2 AND of decisions @ defaults (0.5, 0.5) | 99.6 [99.6,99.6] | 85.5 [84.8,86.1] | 92.0 [91.6,92.4] | 93.4 [92.4,94.3] | 92.7 [92.0,93.4] | 92.3 [91.6,93.0] | 91.9 [91.1,92.6] | 90.3 [89.4,91.2] | +| 3 mean of probs (w=.5) @ defaults (0.5, 0.5) | 99.3 [99.3,99.4] | 88.2 [87.6,88.8] | 93.4 [93.1,93.7] | 94.4 [93.5,95.2] | 93.8 [93.1,94.4] | 93.6 [92.9,94.2] | 93.3 [92.6,93.9] | 92.6 [91.9,93.3] | +| 4a max of probs @ defaults (0.5,) | 97.6 [97.3,97.8] | 92.1 [91.7,92.5] | 94.8 [94.5,95.0] | 95.4 [94.6,96.1] | 95.1 [94.6,95.6] | 94.8 [94.2,95.3] | 94.7 [94.1,95.2] | 94.2 [93.7,94.6] | +| 4b min of probs @ defaults (0.5,) | 99.6 [99.6,99.6] | 85.5 [84.8,86.1] | 92.0 [91.6,92.4] | 93.4 [92.4,94.3] | 92.7 [92.0,93.4] | 92.3 [91.6,93.0] | 91.9 [91.1,92.6] | 90.3 [89.4,91.2] | +| 6 two-stage (Silero + head extend/fill) @ defaults (0.5, 0.5, 16, 16, 30) | 98.9 [98.8,98.9] | 91.2 [90.7,91.7] | 94.9 [94.6,95.1] | 95.4 [94.6,96.1] | 95.1 [94.5,95.6] | 95.0 [94.5,95.5] | 94.9 [94.3,95.4] | 94.3 [93.6,94.8] | +| 7 noise-switch (estimated SNR) @ defaults (0.5, 0.5, 12) | 98.4 [98.1,98.6] | 88.5 [87.9,89.1] | 93.2 [92.8,93.5] | 93.5 [92.5,94.4] | 92.8 [92.0,93.5] | 92.4 [91.6,93.1] | 93.2 [92.4,93.9] | 94.1 [93.6,94.6] | +| 5a logistic reg, 2 probs @ thr .5 | 95.3 [94.9,95.7] | 95.0 [94.6,95.4] | 95.1 [94.9,95.4] | 96.0 [95.2,96.6] | 95.6 [95.1,96.1] | 95.3 [94.8,95.8] | 95.0 [94.6,95.4] | 94.3 [93.8,94.7] | +| 5b logistic reg, + prev/next frame @ thr .5 | 96.1 [95.8,96.4] | 95.7 [95.3,96.1] | 95.9 [95.7,96.1] | 96.5 [95.7,97.1] | 96.2 [95.7,96.7] | 96.1 [95.7,96.5] | 95.9 [95.5,96.2] | 95.2 [94.8,95.6] | +| 5c gradient boosting, + prev/next frame @ thr .5 | 97.0 [96.8,97.2] | 95.6 [95.2,96.0] | 96.3 [96.1,96.5] | 96.8 [96.2,97.3] | 96.5 [96.1,96.9] | 96.3 [95.9,96.7] | 96.2 [95.8,96.6] | 95.9 [95.5,96.3] | +| 5d as 5b, trained also on noise-only clips @ thr .5 | 96.9 [96.7,97.1] | 94.9 [94.4,95.3] | 95.9 [95.7,96.1] | 96.3 [95.5,97.0] | 96.1 [95.6,96.6] | 96.0 [95.6,96.4] | 95.9 [95.5,96.2] | 95.4 [94.9,95.8] | +| 5e as 5c, trained also on noise-only clips @ thr .5 | 97.0 [96.8,97.2] | 95.4 [95.0,95.8] | 96.2 [96.0,96.4] | 96.9 [96.3,97.4] | 96.5 [96.1,96.8] | 96.3 [95.9,96.7] | 96.1 [95.7,96.5] | 95.6 [95.1,96.0] | +| 1 OR of decisions tuned (CV) | 97.7 [97.5,97.9] | 93.0 [92.5,93.5] | 95.3 [95.1,95.5] | 95.6 [94.7,96.3] | 95.2 [94.6,95.7] | 95.3 [94.7,95.8] | 95.4 [94.8,95.8] | 95.3 [94.9,95.6] | +| 2 AND of decisions tuned (CV) | 98.4 [98.2,98.5] | 90.9 [90.3,91.4] | 94.5 [94.2,94.7] | 94.7 [93.8,95.5] | 94.3 [93.7,94.9] | 94.6 [93.9,95.1] | 94.6 [94.0,95.1] | 94.3 [93.7,94.8] | +| 4a max of probs tuned (CV) | 96.8 [96.5,97.2] | 92.9 [92.5,93.3] | 94.8 [94.6,95.1] | 95.6 [94.8,96.2] | 95.2 [94.7,95.7] | 94.9 [94.4,95.5] | 94.6 [94.0,95.1] | 94.2 [93.7,94.7] | +| 4b min of probs tuned (CV) | 98.4 [98.2,98.5] | 90.9 [90.3,91.4] | 94.5 [94.2,94.7] | 94.7 [93.8,95.5] | 94.3 [93.7,94.9] | 94.6 [93.9,95.1] | 94.6 [94.0,95.1] | 94.3 [93.7,94.8] | +| 6 two-stage (Silero + head extend/fill) tuned (CV) | 98.0 [97.8,98.1] | 93.9 [93.5,94.3] | 95.9 [95.7,96.1] | 95.9 [95.1,96.5] | 95.6 [95.1,96.0] | 95.8 [95.4,96.3] | 96.1 [95.8,96.5] | 96.1 [95.7,96.5] | +| 7 noise-switch (estimated SNR) tuned (CV) | 97.6 [97.3,97.9] | 91.9 [91.4,92.3] | 94.7 [94.4,94.9] | 94.7 [93.8,95.4] | 95.1 [94.5,95.6] | 94.8 [94.2,95.3] | 94.6 [94.1,95.1] | 94.1 [93.6,94.6] | +| 3 mean of probs (w=.5) tuned w,thr (CV) | 97.1 [96.8,97.4] | 93.6 [93.2,94.1] | 95.3 [95.1,95.6] | 95.7 [94.9,96.4] | 95.3 [94.8,95.8] | 95.3 [94.8,95.8] | 95.3 [94.9,95.8] | 95.2 [94.7,95.6] | +| 5a logistic reg, 2 probs tuned thr (CV) | 96.6 [96.3,96.9] | 94.0 [93.5,94.4] | 95.3 [95.0,95.5] | 95.7 [94.9,96.4] | 95.4 [94.8,95.9] | 95.4 [94.8,95.9] | 95.3 [94.9,95.7] | 94.8 [94.3,95.2] | +| 5b logistic reg, + prev/next frame tuned thr (CV) | 97.0 [96.8,97.3] | 95.0 [94.6,95.4] | 96.0 [95.8,96.2] | 96.4 [95.6,97.0] | 96.1 [95.6,96.5] | 96.2 [95.8,96.6] | 96.0 [95.6,96.4] | 95.5 [95.1,95.9] | +| 5c gradient boosting, + prev/next frame tuned thr (CV) | 96.9 [96.7,97.1] | 95.7 [95.3,96.0] | 96.3 [96.1,96.5] | 96.8 [96.3,97.3] | 96.5 [96.1,96.9] | 96.3 [95.9,96.7] | 96.2 [95.8,96.5] | 95.9 [95.5,96.2] | +| 5d as 5b, trained also on noise-only clips tuned thr (CV) | 96.9 [96.7,97.1] | 94.9 [94.4,95.3] | 95.9 [95.7,96.1] | 96.3 [95.5,97.0] | 96.1 [95.6,96.6] | 96.0 [95.6,96.4] | 95.9 [95.5,96.2] | 95.4 [94.9,95.8] | +| 5e as 5c, trained also on noise-only clips tuned thr (CV) | 96.9 [96.7,97.1] | 95.5 [95.1,95.9] | 96.2 [96.0,96.4] | 96.9 [96.3,97.4] | 96.5 [96.1,96.8] | 96.3 [95.9,96.7] | 96.1 [95.7,96.5] | 95.6 [95.1,96.0] | +| 7o oracle noise-switch (true SNR) tuned (CV) | 97.8 [97.5,98.0] | 90.9 [90.4,91.4] | 94.2 [93.9,94.5] | 94.0 [93.0,94.9] | 93.4 [92.7,94.0] | 94.8 [94.2,95.3] | 94.6 [94.1,95.1] | 94.1 [93.6,94.6] | + +### False alarms on noise-only clips + +noise-only clips (12 x 30 s per level, white+pink pooled over the levels shown). False-alarm segments per hour (frame false-positive rate in brackets). + +| system | 20 dB | 10 dB | 5 dB | 0 dB | all | +|---|---|---|---|---|---| +| Silero thr .5 (unified post) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | +| Silero thr .5 (native defaults: min speech .25, pause .1, pad .03) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | +| redux head thr .5 | 485 (95.43%) | 195 (98.52%) | 170 (98.70%) | 170 (98.68%) | 255 (97.83%) | +| Silero tuned thr (CV) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | +| redux head tuned thr (CV) | 255 (97.76%) | 135 (99.01%) | 135 (99.03%) | 135 (99.03%) | 165 (98.71%) | +| 1 OR of decisions @ defaults (0.5, 0.5) | 485 (95.43%) | 195 (98.52%) | 170 (98.70%) | 170 (98.68%) | 255 (97.83%) | +| 2 AND of decisions @ defaults (0.5, 0.5) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | +| 3 mean of probs (w=.5) @ defaults (0.5, 0.5) | 0 (0.00%) | 5 (0.03%) | 5 (0.04%) | 10 (0.08%) | 5 (0.04%) | +| 4a max of probs @ defaults (0.5,) | 485 (95.43%) | 195 (98.52%) | 170 (98.70%) | 170 (98.68%) | 255 (97.83%) | +| 4b min of probs @ defaults (0.5,) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | +| 6 two-stage (Silero + head extend/fill) @ defaults (0.5, 0.5, 16, 16, 30) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | +| 7 noise-switch (estimated SNR) @ defaults (0.5, 0.5, 12) | 485 (95.43%) | 195 (98.52%) | 170 (98.70%) | 170 (98.68%) | 255 (97.83%) | +| 5a logistic reg, 2 probs @ thr .5 | 370 (2.73%) | 960 (13.42%) | 730 (14.70%) | 475 (10.07%) | 634 (10.23%) | +| 5b logistic reg, + prev/next frame @ thr .5 | 110 (0.78%) | 525 (5.73%) | 495 (8.33%) | 300 (6.14%) | 358 (5.24%) | +| 5c gradient boosting, + prev/next frame @ thr .5 | 2160 (39.14%) | 2005 (62.50%) | 1960 (67.37%) | 1800 (70.39%) | 1981 (59.85%) | +| 5d as 5b, trained also on noise-only clips @ thr .5 | 0 (0.00%) | 5 (0.02%) | 10 (0.09%) | 10 (0.07%) | 6 (0.05%) | +| 5e as 5c, trained also on noise-only clips @ thr .5 | 0 (0.00%) | 25 (0.11%) | 35 (0.24%) | 15 (0.14%) | 19 (0.12%) | +| 1 OR of decisions tuned (CV) | 1415 (82.51%) | 645 (94.09%) | 515 (94.99%) | 490 (95.38%) | 766 (91.74%) | +| 2 AND of decisions tuned (CV) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | +| 4a max of probs tuned (CV) | 255 (97.76%) | 135 (99.01%) | 135 (99.03%) | 135 (99.03%) | 165 (98.71%) | +| 4b min of probs tuned (CV) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | +| 6 two-stage (Silero + head extend/fill) tuned (CV) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | +| 7 noise-switch (estimated SNR) tuned (CV) | 485 (95.43%) | 195 (98.52%) | 170 (98.70%) | 170 (98.68%) | 255 (97.83%) | +| 3 mean of probs (w=.5) tuned w,thr (CV) | 1350 (83.67%) | 555 (94.87%) | 475 (95.49%) | 460 (95.73%) | 710 (92.44%) | +| 5a logistic reg, 2 probs tuned thr (CV) | 115 (0.67%) | 645 (5.86%) | 620 (7.87%) | 385 (5.58%) | 441 (4.99%) | +| 5b logistic reg, + prev/next frame tuned thr (CV) | 30 (0.13%) | 275 (2.27%) | 350 (3.73%) | 235 (3.02%) | 222 (2.29%) | +| 5c gradient boosting, + prev/next frame tuned thr (CV) | 2160 (39.64%) | 1985 (62.81%) | 1950 (67.77%) | 1805 (70.68%) | 1975 (60.23%) | +| 5d as 5b, trained also on noise-only clips tuned thr (CV) | 0 (0.00%) | 5 (0.02%) | 10 (0.09%) | 10 (0.07%) | 6 (0.05%) | +| 5e as 5c, trained also on noise-only clips tuned thr (CV) | 5 (0.02%) | 25 (0.11%) | 35 (0.25%) | 15 (0.14%) | 20 (0.13%) | +| 7o oracle noise-switch (true SNR) tuned (CV) | 0 (0.00%) | 195 (98.52%) | 170 (98.70%) | 170 (98.68%) | 134 (73.98%) | + +**Best single (CV-tuned):** redux head tuned thr (CV). + +| fusion (CV) | dF1 vs best single, pooled | dF1 5 dB | dF1 0 dB | dR pooled | dP pooled | +|---|---|---|---|---|---| +| 1 OR of decisions @ defaults (0.5, 0.5) | -0.02 [-0.08,+0.04] | +0.11 [-0.03,+0.27] | -0.01 [-0.17,+0.16] | -0.69 [-0.79,-0.60] | +0.71 [+0.61,+0.83] | +| 2 AND of decisions @ defaults (0.5, 0.5) | -2.79 [-3.02,-2.56] | -2.67 [-3.15,-2.20] | -3.87 [-4.57,-3.19] | -7.36 [-7.82,-6.90] | +2.77 [+2.43,+3.13] | +| 3 mean of probs (w=.5) @ defaults (0.5, 0.5) | -1.35 [-1.53,-1.16] | -1.25 [-1.66,-0.82] | -1.53 [-2.06,-1.01] | -4.61 [-4.95,-4.28] | +2.49 [+2.17,+2.85] | +| 4a max of probs @ defaults (0.5,) | -0.02 [-0.08,+0.04] | +0.11 [-0.03,+0.27] | -0.01 [-0.17,+0.16] | -0.69 [-0.79,-0.60] | +0.71 [+0.61,+0.83] | +| 4b min of probs @ defaults (0.5,) | -2.79 [-3.02,-2.56] | -2.67 [-3.15,-2.20] | -3.87 [-4.57,-3.19] | -7.36 [-7.82,-6.90] | +2.77 [+2.43,+3.13] | +| 6 two-stage (Silero + head extend/fill) @ defaults (0.5, 0.5, 16, 16, 30) | +0.09 [-0.04,+0.22] | +0.33 [+0.03,+0.66] | +0.09 [-0.36,+0.54] | -1.60 [-1.88,-1.35] | +2.00 [+1.69,+2.34] | +| 7 noise-switch (estimated SNR) @ defaults (0.5, 0.5, 12) | -1.62 [-1.81,-1.42] | -1.35 [-1.89,-0.84] | -0.04 [-0.21,+0.12] | -4.32 [-4.70,-3.95] | +1.52 [+1.34,+1.74] | +| 5a logistic reg, 2 probs @ thr .5 | +0.36 [+0.22,+0.49] | +0.42 [+0.05,+0.80] | +0.13 [-0.32,+0.57] | +2.17 [+1.95,+2.38] | -1.54 [-1.80,-1.28] | +| 5b logistic reg, + prev/next frame @ thr .5 | +1.14 [+1.01,+1.28] | +1.34 [+0.98,+1.70] | +1.05 [+0.63,+1.45] | +2.90 [+2.71,+3.09] | -0.71 [-0.94,-0.44] | +| 5c gradient boosting, + prev/next frame @ thr .5 | +1.50 [+1.37,+1.64] | +1.63 [+1.31,+1.98] | +1.73 [+1.40,+2.08] | +2.79 [+2.59,+2.98] | +0.13 [-0.16,+0.44] | +| 5d as 5b, trained also on noise-only clips @ thr .5 | +1.10 [+0.96,+1.25] | +1.29 [+0.98,+1.63] | +1.20 [+0.77,+1.62] | +2.09 [+1.84,+2.33] | +0.05 [-0.24,+0.38] | +| 5e as 5c, trained also on noise-only clips @ thr .5 | +1.42 [+1.29,+1.57] | +1.57 [+1.26,+1.90] | +1.40 [+1.03,+1.77] | +2.59 [+2.34,+2.82] | +0.18 [-0.11,+0.51] | +| 1 OR of decisions tuned (CV) | +0.53 [+0.42,+0.65] | +0.80 [+0.52,+1.09] | +1.09 [+0.84,+1.37] | +0.23 [+0.03,+0.41] | +0.85 [+0.63,+1.11] | +| 2 AND of decisions tuned (CV) | -0.32 [-0.46,-0.16] | +0.05 [-0.29,+0.42] | +0.11 [-0.32,+0.54] | -1.93 [-2.23,-1.65] | +1.50 [+1.22,+1.84] | +| 4a max of probs tuned (CV) | +0.04 [+0.03,+0.05] | +0.04 [+0.02,+0.06] | +0.04 [+0.01,+0.06] | +0.09 [+0.06,+0.11] | -0.02 [-0.02,-0.01] | +| 4b min of probs tuned (CV) | -0.32 [-0.46,-0.16] | +0.05 [-0.29,+0.42] | +0.11 [-0.32,+0.54] | -1.93 [-2.23,-1.65] | +1.50 [+1.22,+1.84] | +| 6 two-stage (Silero + head extend/fill) tuned (CV) | +1.13 [+1.00,+1.28] | +1.56 [+1.27,+1.89] | +1.94 [+1.56,+2.34] | +1.12 [+0.94,+1.29] | +1.14 [+0.88,+1.44] | +| 7 noise-switch (estimated SNR) tuned (CV) | -0.14 [-0.21,-0.06] | +0.07 [-0.07,+0.24] | -0.04 [-0.21,+0.12] | -0.94 [-1.06,-0.82] | +0.75 [+0.65,+0.88] | +| 3 mean of probs (w=.5) tuned w,thr (CV) | +0.55 [+0.47,+0.64] | +0.78 [+0.59,+0.97] | +0.99 [+0.75,+1.23] | +0.82 [+0.67,+0.97] | +0.25 [+0.11,+0.42] | +| 5a logistic reg, 2 probs tuned thr (CV) | +0.48 [+0.36,+0.61] | +0.75 [+0.46,+1.04] | +0.63 [+0.23,+1.02] | +1.14 [+0.94,+1.33] | -0.23 [-0.46,+0.04] | +| 5b logistic reg, + prev/next frame tuned thr (CV) | +1.21 [+1.10,+1.34] | +1.48 [+1.19,+1.79] | +1.35 [+1.01,+1.69] | +2.19 [+2.00,+2.37] | +0.17 [-0.07,+0.46] | +| 5c gradient boosting, + prev/next frame tuned thr (CV) | +1.51 [+1.38,+1.66] | +1.63 [+1.30,+1.98] | +1.72 [+1.40,+2.06] | +2.89 [+2.69,+3.08] | +0.05 [-0.24,+0.36] | +| 5d as 5b, trained also on noise-only clips tuned thr (CV) | +1.10 [+0.96,+1.25] | +1.29 [+0.98,+1.63] | +1.20 [+0.77,+1.62] | +2.09 [+1.84,+2.33] | +0.05 [-0.24,+0.38] | +| 5e as 5c, trained also on noise-only clips tuned thr (CV) | +1.42 [+1.28,+1.56] | +1.55 [+1.24,+1.87] | +1.39 [+1.00,+1.75] | +2.71 [+2.46,+2.94] | +0.05 [-0.24,+0.37] | + +### Noise-shift split (redux): parameters chosen on clean/20/10 dB clips only, scored on 5 and 0 dB clips + +| system | P | R | F1 (5+0 dB clips) | +|---|---|---|---| +| Silero (0.1,) | 97.6 [97.4,97.9] | 91.5 [90.6,92.2] | 94.5 [94.0,94.8] | +| redux head (0.3,) | 90.0 [89.0,90.9] | 97.2 [96.8,97.5] | 93.4 [93.0,93.8] | +| OR (0.1, 0.4) | 92.8 [92.0,93.4] | 96.3 [95.9,96.7] | 94.5 [94.1,94.9] | +| AND (0.1, 0.1) | 97.7 [97.4,97.9] | 91.4 [90.5,92.2] | 94.4 [94.0,94.8] | +| mean (0.75, 0.1) | 90.2 [89.4,91.0] | 97.9 [97.6,98.2] | 93.9 [93.5,94.3] | +| max (0.3,) | 89.9 [88.9,90.7] | 97.3 [96.9,97.6] | 93.5 [93.0,93.9] | +| min (0.1,) | 97.7 [97.4,97.9] | 91.4 [90.6,92.2] | 94.4 [94.0,94.8] | +| two-stage (0.3, 0.3, 32.0, 16.0, 60.0) | 97.5 [97.2,97.7] | 94.8 [94.3,95.3] | 96.1 [95.9,96.4] | +| noise-switch (0.3, 0.3, 40.0) | 90.0 [89.0,90.8] | 97.2 [96.8,97.5] | 93.4 [93.0,93.8] | +| lr2 (0.55,) | 92.2 [91.6,92.7] | 96.9 [96.6,97.2] | 94.5 [94.2,94.8] | +| lr6 (0.55,) | 93.7 [93.2,94.2] | 97.2 [96.9,97.5] | 95.4 [95.1,95.7] | +| hgb6 (0.45,) | 94.9 [94.4,95.3] | 96.8 [96.3,97.2] | 95.8 [95.6,96.1] | +| lr6nz (0.45,) | 94.4 [94.0,94.8] | 96.4 [95.9,96.8] | 95.4 [95.1,95.7] | +| hgb6nz (0.5,) | 95.6 [95.2,96.0] | 95.9 [95.3,96.4] | 95.8 [95.5,96.0] | + +### Operating points with precision >= 99% (redux): CV-selected maximum recall + +| system | P | R | F1 | chosen params per fold | +|---|---|---|---|---| +| Silero | 99.1 [99.0,99.2] | 88.6 [88.0,89.2] | 93.6 [93.2,93.9] | [(0.2,), (0.2,), (0.2,), (0.25,)] | +| redux head | 99.1 [99.0,99.1] | 89.4 [88.9,89.9] | 94.0 [93.7,94.3] | [(0.65,), (0.7,), (0.7,), (0.75,)] | +| OR | 98.9 [98.8,99.0] | 89.9 [89.3,90.4] | 94.2 [93.9,94.5] | [(0.3, 0.7), (0.2, 0.8), (0.5, 0.7), (0.4, 0.8)] | +| AND | 99.0 [99.0,99.1] | 89.2 [88.6,89.7] | 93.9 [93.6,94.2] | [(0.1, 0.4), (0.1, 0.5), (0.1, 0.5), (0.2, 0.2)] | +| mean | 98.9 [98.8,99.0] | 90.0 [89.4,90.5] | 94.3 [94.0,94.5] | [(0.25, 0.5), (0.75, 0.25), (0.25, 0.55), (0.5, 0.45)] | +| max | 99.0 [98.9,99.1] | 89.6 [89.0,90.1] | 94.1 [93.8,94.3] | [(0.65,), (0.7,), (0.7,), (0.75,)] | +| min | 99.1 [99.1,99.2] | 88.7 [88.1,89.2] | 93.6 [93.3,93.9] | [(0.2,), (0.2,), (0.2,), (0.2,)] | +| two-stage | 99.0 [98.9,99.1] | 91.5 [90.9,92.0] | 95.1 [94.8,95.3] | [(0.5, 0.3, 32.0, 0.0, 60.0), (0.7, 0.3, 16.0, 8.0, 60.0), (0.5, 0.3, 16.0, 0.0, 60.0), (0.5, 0.3, 16.0, 0.0, 60.0)] | +| noise-switch | 99.1 [99.0,99.2] | 89.3 [88.8,89.8] | 94.0 [93.7,94.2] | [(0.3, 0.7, 40.0), (0.3, 0.7, 40.0), (0.3, 0.7, 40.0), (0.5, 0.7, 28.0)] | +| lr2 | 99.1 [99.1,99.2] | 89.4 [88.9,90.0] | 94.0 [93.7,94.3] | thr [0.75, 0.8, 0.8, 0.75] | +| lr6 | 99.1 [99.0,99.2] | 90.7 [90.1,91.2] | 94.7 [94.4,95.0] | thr [0.8, 0.8, 0.8, 0.8] | +| hgb6 | 99.0 [98.9,99.1] | 91.9 [91.4,92.4] | 95.3 [95.0,95.6] | thr [0.8, 0.85, 0.8, 0.8] | +| lr6nz | 99.0 [99.0,99.1] | 90.6 [90.0,91.2] | 94.6 [94.3,94.9] | thr [0.8, 0.8, 0.8, 0.8] | +| hgb6nz | 99.0 [98.9,99.1] | 91.9 [91.3,92.3] | 95.3 [95.0,95.5] | thr [0.8, 0.85, 0.8, 0.8] | + +### In-sample best operating points (redux), descriptive only (selected on all clips, optimistic) + +| rule | best F1 params | P | R | F1 | max R at P>=99% params | P | R | +|---|---|---|---|---|---|---|---| +| silero | (0.1,) | 98.3 | 91.1 | 94.5 | (0.2,) | 99.1 | 88.8 | +| head | (0.45,) | 96.9 | 92.8 | 94.8 | (0.7,) | 99.1 | 89.5 | +| or | (0.1, 0.6) | 97.7 | 93.0 | 95.3 | (0.4, 0.7) | 99.0 | 90.0 | +| and | (0.1, 0.1) | 98.4 | 90.9 | 94.5 | (0.1, 0.5) | 99.1 | 89.2 | +| mean | (0.75, 0.15) | 97.1 | 93.6 | 95.3 | (0.5, 0.4) | 99.0 | 90.2 | +| max | (0.45,) | 96.8 | 92.9 | 94.8 | (0.7,) | 99.1 | 89.6 | +| min | (0.1,) | 98.4 | 90.9 | 94.5 | (0.2,) | 99.1 | 88.7 | +| two | (0.3, 0.3, 32.0, 16.0, 60.0) | 98.0 | 93.9 | 95.9 | (0.3, 0.3, 8.0, 0.0, 60.0) | 99.0 | 91.5 | +| switch | (0.3, 0.5, 40.0) | 97.6 | 91.9 | 94.7 | (0.3, 0.7, 40.0) | 99.1 | 89.4 | +| oracle | (0.3, 0.5, 10.0) | 97.8 | 90.9 | 94.2 | (0.3, 0.7, 10.0) | 99.2 | 88.9 | + +LR coefficients (redux, fit on folds != 0; features lg(silero), lg(head)[, lg(silero prev/next), lg(head prev/next)]): +- lr2: ([0.457, 0.322], 1.7936439121924497) +- lr6: ([0.108, -0.552, 0.096, 0.31, 0.371, 0.545], 1.7755007987974973) +- lr6nz: ([0.026, -0.684, 0.353, 0.543, 0.25, 0.56], 2.5562606732085627) diff --git a/scripts/vad_bench/fusion/rules.py b/scripts/vad_bench/fusion/rules.py new file mode 100644 index 0000000..d21490e --- /dev/null +++ b/scripts/vad_bench/fusion/rules.py @@ -0,0 +1,49 @@ +import numpy as np +from fl import * +THR=[round(x,2) for x in np.arange(0.1,0.91,0.05)] +THR9=[round(x,1) for x in np.arange(0.1,0.91,0.1)] +def dilate(m,pre,post): + """grow each run: `pre` cells earlier, `post` cells later""" + if pre==0 and post==0: return m + s,e=runs(m); out=m.copy(); n=len(m) + for a,b in zip(s,e): out[max(0,a-pre):a]=True; out[b:min(n,b+post)]=True + return out +def two_stage(S,H,pre,post,Y): + """Silero decides; head extends boundaries by up to pre/post cells (only where head says speech, contiguous to the silero run) + and fills silero gaps shorter than Y cells when the head calls >=50% of the gap speech.""" + out=S.copy(); n=len(S) + if pre or post: + s,e=runs(S) + for a,b in zip(s,e): + # walk outward while head is speech, max pre/post cells + i=a + while i>0 and a-i
0 and b=0.5: out[a:b]=True
+    return out
+def raw_mask(rule,p,d,h):
+    s=d["silero"];hh=d[h]
+    if rule=="silero": return s>=p[0]
+    if rule=="head": return hh>=p[0]
+    if rule=="or": return (s>=p[0])|(hh>=p[1])
+    if rule=="and": return (s>=p[0])&(hh>=p[1])
+    if rule=="mean": w,t=p; return (w*s+(1-w)*hh)>=t
+    if rule=="max": return np.maximum(s,hh)>=p[0]
+    if rule=="min": return np.minimum(s,hh)>=p[0]
+    if rule=="two": ts,th,pre,post,Y=p; return two_stage(s>=ts,hh>=th,pre,post,Y)
+    if rule=="switch": ts,th,T=p; return (hh>=th) if d["snr_est"]=ts)
+    if rule=="oracle": ts,th,T=p; return (hh>=th) if d["snr_true"]<=T else (s>=ts)
+    raise ValueError(rule)
+def candidates(rule):
+    if rule in("silero","head","max","min"): return [(t,) for t in THR]
+    if rule in("or","and"): return [(a,b) for a in THR9 for b in THR9]
+    if rule=="mean": return [(w,t) for w in(0.25,0.5,0.75) for t in THR]
+    if rule=="two": return [(ts,th,pre,post,Y) for ts in(.3,.5,.7) for th in(.3,.5,.7) for pre in(0,8,16,32,48) for post in(0,8,16,32) for Y in(0,15,30,60)]
+    if rule=="switch": return [(ts,th,T) for ts in(.3,.5,.7) for th in(.3,.5,.7) for T in(8,12,16,20,24,28,32,40)]
+    if rule=="oracle": return [(ts,th,T) for ts in(.3,.5,.7) for th in(.3,.5,.7) for T in(5,10)]
diff --git a/scripts/vad_bench/fusion/run.py b/scripts/vad_bench/fusion/run.py
new file mode 100644
index 0000000..2eccdae
--- /dev/null
+++ b/scripts/vad_bench/fusion/run.py
@@ -0,0 +1,58 @@
+import sys,pickle,time,numpy as np
+from multiprocessing import Pool
+from fl import *
+from rules import *
+from sklearn.linear_model import LogisticRegression
+from sklearn.ensemble import HistGradientBoostingClassifier
+D=load()
+for i,d in enumerate(D):
+    c=d["cond"].replace("nz-",""); d["snr_true"]=99 if c=="clean" else int(c.lstrip("whitepink"))
+    d["noise_only"]=d["name"].startswith("nz-")
+    k=int(d["name"].split("_")[1]); d["fold"]=k%4 if d["noise_only"] else d["fold"]
+N=len(D)
+def ev(mask,d,pk):
+    m,regs=post(mask,**pk)
+    c=counts(m,d["ref_mask"]); return np.append(c,len(regs))
+def eval_cfg(a):
+    rule,p,h,pk=a
+    return np.array([ev(raw_mask(rule,p,d,h),d,pk) for d in D])
+def lg(x): x=np.clip(x,1e-4,1-1e-4); return np.log(x/(1-x))
+def feats(d,h,kind):
+    s=d["silero"];hh=d[h]
+    if kind=="lr2": return np.stack([lg(s),lg(hh)],1)
+    def sh(x,k): return np.concatenate([np.full(k,x[0]),x[:-k]]) if k>0 else np.concatenate([x[-k:],np.full(-k,x[-1])])
+    return np.stack([lg(s),lg(hh),lg(sh(s,3)),lg(sh(s,-3)),lg(sh(hh,8)),lg(sh(hh,-8))],1)
+def lr_split(a):
+    h,kind,sname,train=a
+    nz=kind.endswith("nz"); kind=kind[:-2] if nz else kind
+    if not nz: train=[i for i in train if not D[i]["noise_only"]]
+    X=np.concatenate([feats(D[i],h,kind)[::10] for i in train]); y=np.concatenate([D[i]["ref_mask"][::10] for i in train])
+    mdl=HistGradientBoostingClassifier(max_iter=120,learning_rate=0.1,random_state=0) if kind=="hgb6" else LogisticRegression(C=1.0,max_iter=300)
+    mdl.fit(X,y)
+    P=[mdl.predict_proba(feats(d,h,kind))[:,1] for d in D]
+    out=np.zeros((len(THR),N,5))
+    for ti,t in enumerate(THR):
+        for i,d in enumerate(D): out[ti,i]=ev(P[i]>=t,d,POST_HEAD)
+    coef=getattr(mdl,"coef_",None)
+    return (h,kind+("nz" if nz else ""),sname),out,(None if coef is None else (coef[0].round(3).tolist(),float(mdl.intercept_[0])))
+if __name__=="__main__":
+    t0=time.time(); res={}
+    jobs=[]
+    for h in("ultra","redux"):
+        for rule in("head","or","and","mean","max","min","two","switch","oracle"):
+            for p in candidates(rule): jobs.append((rule,p,h,POST_HEAD))
+    for p in candidates("silero"): jobs.append(("silero",p,"ultra",POST_HEAD))
+    # native default post for singles at thr 0.5
+    jobs.append(("silero",(0.5,),"ultra",POST_SIL))
+    print(len(jobs),"jobs",flush=True)
+    with Pool(5) as pool:
+        outs=pool.map(eval_cfg,jobs,chunksize=4)
+    res["cfg"]={ (j[0],j[1],j[2],j[3]["min_speech"]):o for j,o in zip(jobs,outs)}
+    print("rules done",time.time()-t0,flush=True)
+    folds=np.array([d["fold"] for d in D]); easy=np.array([d["snr_true"]>=10 for d in D])
+    splits=[(f"f{f}",[i for i in range(N) if folds[i]!=f]) for f in range(4)]+[("shift",[i for i in range(N) if easy[i]])]
+    lj=[(h,k,sn,tr) for h in("ultra","redux") for k in("lr2","lr6","hgb6","lr6nz","hgb6nz") for sn,tr in splits]
+    with Pool(5) as pool: lo=pool.map(lr_split,lj,chunksize=1)
+    res["lr"]={k:(o,c) for k,o,c in lo}
+    res["meta"]=dict(names=[d["name"] for d in D],cond=[d["cond"] for d in D],fold=folds.tolist(),snr_true=[d["snr_true"] for d in D],noise_only=[d["noise_only"] for d in D],snr_est=[d["snr_est"] for d in D],dur=[d["dur"] for d in D])
+    pickle.dump(res,open("results.pkl","wb")); print("done",time.time()-t0)
diff --git a/scripts/vad_bench/fusion/ted.py b/scripts/vad_bench/fusion/ted.py
new file mode 100644
index 0000000..ecd6e3b
--- /dev/null
+++ b/scripts/vad_bench/fusion/ted.py
@@ -0,0 +1,23 @@
+import sys,json,numpy as np,soundfile as sf,subprocess
+import collect
+from run import *
+f=sys.argv[1]   # 16 kHz mono wav of one long talk
+import onnxruntime as ort
+so=ort.SessionOptions(); so.intra_op_num_threads=1; so.inter_op_num_threads=1
+sess=ort.InferenceSession(collect.M,so,providers=["CPUExecutionProvider"])
+y=sf.read(f,dtype="float32")[0]; n=int(round(len(y)/16000/GR))
+d=dict(n=n,silero=hold(collect.sil(y,sess),0.032,n),snr_est=0.0)
+for k,g in collect.HEADS.items():
+    j=json.loads(subprocess.run([collect.CLI,"vad","--model",g,"--input",f,"--probabilities","--threads","1"],capture_output=True,text=True,check=True).stdout)
+    d[k]=hold(np.array(j["probabilities"],np.float32),0.08,n)
+tr=[i for i,x in enumerate(D) if not x["noise_only"]]
+def agree(a,b):
+    c=counts(a,b); tp,fp,fn,tn=c; P,R,F=prf(c); po=(tp+tn)/c.sum(); pe=((tp+fp)*(tp+fn)+(fn+tn)*(fp+tn))/c.sum()**2; return F,(po-pe)/(1-pe)
+for h in("ultra","redux"):
+    X=np.concatenate([feats(D[i],h,"lr6")[::10] for i in tr]); yy=np.concatenate([D[i]["ref_mask"][::10] for i in tr])
+    mdl=LogisticRegression(C=1.0,max_iter=300).fit(X,yy)
+    ms={"Silero .5":raw_mask("silero",(0.5,),d,h),f"{h} head .5":raw_mask("head",(0.5,),d,h),"OR":raw_mask("or",(0.5,0.5),d,h),"AND":raw_mask("and",(0.5,0.5),d,h),"mean .5":raw_mask("mean",(0.5,0.5),d,h),"two-stage .5/.5/160/160/300":raw_mask("two",(0.5,0.5,16,16,30),d,h),"LR+ctx @.5 (fit on synthetic)":mdl.predict_proba(feats(d,h,"lr6"))[:,1]>=0.5}
+    ms={k:post(v,**POST_HEAD)[0] for k,v in ms.items()}
+    print(f"## TED talk {len(y)/16000:.0f} s, head={h}: speech fraction and agreement (F1/kappa) with Silero and with the head")
+    for k,v in ms.items():
+        a=agree(v,ms["Silero .5"]); b=agree(v,ms[f"{h} head .5"]); print(f"| {k} | {100*v.mean():.1f}% | vs Silero {100*a[0]:.1f}/{a[1]:.3f} | vs head {100*b[0]:.1f}/{b[1]:.3f} |")
diff --git a/scripts/vad_bench/fusion/ted_results.txt b/scripts/vad_bench/fusion/ted_results.txt
new file mode 100644
index 0000000..dc9259b
--- /dev/null
+++ b/scripts/vad_bench/fusion/ted_results.txt
@@ -0,0 +1,16 @@
+## TED talk 600 s, head=ultra: speech fraction and agreement (F1/kappa) with Silero and with the head
+| Silero .5 | 77.5% | vs Silero 100.0/1.000 | vs head 95.6/0.777 |
+| ultra head .5 | 83.4% | vs Silero 95.6/0.777 | vs head 100.0/1.000 |
+| OR | 84.1% | vs Silero 95.9/0.788 | vs head 99.6/0.975 |
+| AND | 76.5% | vs Silero 99.4/0.973 | vs head 95.7/0.785 |
+| mean .5 | 79.7% | vs Silero 98.2/0.915 | vs head 97.3/0.855 |
+| two-stage .5/.5/160/160/300 | 83.8% | vs Silero 96.1/0.799 | vs head 99.4/0.964 |
+| LR+ctx @.5 (fit on synthetic) | 90.5% | vs Silero 92.2/0.531 | vs head 95.6/0.664 |
+## TED talk 600 s, head=redux: speech fraction and agreement (F1/kappa) with Silero and with the head
+| Silero .5 | 77.5% | vs Silero 100.0/1.000 | vs head 95.5/0.769 |
+| redux head .5 | 84.3% | vs Silero 95.5/0.769 | vs head 100.0/1.000 |
+| OR | 84.5% | vs Silero 95.6/0.772 | vs head 99.9/0.991 |
+| AND | 77.2% | vs Silero 99.8/0.991 | vs head 95.6/0.772 |
+| mean .5 | 80.8% | vs Silero 97.7/0.888 | vs head 97.8/0.876 |
+| two-stage .5/.5/160/160/300 | 84.1% | vs Silero 95.9/0.787 | vs head 99.6/0.976 |
+| LR+ctx @.5 (fit on synthetic) | 90.2% | vs Silero 92.4/0.543 | vs head 96.3/0.710 |
diff --git a/scripts/vad_bench/noise_dive/README.md b/scripts/vad_bench/noise_dive/README.md
new file mode 100644
index 0000000..d2fe3ce
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/README.md
@@ -0,0 +1,30 @@
+# Why the Parakeet VAD head fires on noise
+
+Scripts and small results of the root-cause study described in
+[../../../docs/vad-benchmarks.md](../../../docs/vad-benchmarks.md#the-head-on-noise-only-audio).
+No audio, model, `.npz` or `.pkl` file is committed. A re-run regenerates them.
+
+## Inputs (set as environment variables)
+
+| Variable | Meaning |
+| --- | --- |
+| `FUSION_DIR` | `scripts/vad_bench/fusion` with `data/libri.npy` and `data/spk.npy` built by `prep_libri.py` (default `../fusion`) |
+| `NOISE_DIVE_WORK` | working directory with `ref/` (reference weights as `_weights.npz`), `hfmeta/` (`.config.json`) and the CLI build |
+| `PARAKEET_CLI` | path to `parakeet-cli` |
+| `GGUF_ROOT` | directory with `gguf/` and `vad-card/` GGUF files |
+| `SILERO_ONNX` | path to `silero_vad.onnx` (used by `e_build.py`, `e2_talk.py`) |
+
+Run the scripts from this directory. Python packages: numpy, torch, soundfile,
+transformers (with the Parakeet classes), onnxruntime, matplotlib.
+
+## Files
+
+- `vadlib.py`: independent reference of the head path built on Hugging Face transformers.
+  `sig.py` and `corpus.py`: signal generators and the speech clips.
+- `a_faith.py`, `cli_check.py`, `a2_variants.py`: the reference against the CLI; head variants (`res_A*`, `res_cli_check.txt`).
+- `b1` to `b10`: amplitude, signal types, normalisation, regression, level and share of noise (`res_B*`).
+- `d_prep.py`, `d_run.py`, `d_analyze.py`, `d_probs.py`, `d_mit.py`: the 60 s block in a real talk with `transcribe --vad` (`res_D*`).
+  `res_D_silero.txt` holds the Silero segment check, done by hand from its segment file.
+- `e_build.py`, `e_eval.py`, `e2_talk.py`: the mitigations (`res_E*`).
+- `fig_*.png`: four figures (logit histograms, false alarms against level, level and variation of the stretch, offset and scale probe).
+- The `res_*.md` and `res_*.txt` files are the results quoted in the docs.
diff --git a/scripts/vad_bench/noise_dive/a2_variants.py b/scripts/vad_bench/noise_dive/a2_variants.py
new file mode 100644
index 0000000..1f2fe04
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/a2_variants.py
@@ -0,0 +1,28 @@
+"""Does an alternative head wiring (the documented debug variants) change the noise behaviour? (rules out 'wrong activation/residual' as the cause)"""
+import numpy as np,torch,json
+from vadlib import *; from sig import *; import corpus
+F=torch.nn.functional
+def head(r,s,act_proj,res,act_ctx):
+    a=lambda k,x: F.silu(x) if k=='silu' else F.relu(x)
+    h=a(act_proj,s@r.pw.T+r.pb); c=F.conv1d(h.T[None],r.cw,r.cb,padding=2)[0].T
+    if res: c=c+h
+    return (a(act_ctx,c)@r.ow+r.ob).numpy()
+V=[('silu','silu',False),('relu','relu',False),('silu','silu',True),('relu','relu',True)]
+cl=corpus.clean_clips()[:20]; gaps=[g for g in corpus.gap_clips() if g['snr'] in(5,0)][:12:3]+[g for g in corpus.gap_clips() if g['snr'] in(5,0)][48:60:3]
+rng=np.random.default_rng(5); N=30*SR
+nz=[at_db(white(N,rng),-23),at_db(colored(N,rng,1),-23),at_db(white(N,rng),-40),at_db(colored(N,rng,1),-40)]
+for m in ['redux','ultra']:
+    r=ref(m); print(f'\n{m}: variant (proj act, ctx act, ctx residual) | clean speech F1 % | speech frames in noise-only -23 dBFS % | -40 dBFS % | embedded 5/0 dB stretch FP %')
+    sub_cl=[(r.sub_out(norm_mel(raw_logmel(y))),sp) for y,sp in cl]
+    sub_nz=[r.sub_out(norm_mel(raw_logmel(x))) for x in nz]
+    sub_g=[(r.sub_out(norm_mel(raw_logmel(g['y']))),g) for g in gaps]
+    for ap,ac,res in V:
+        c=np.zeros(3)
+        for s,sp in sub_cl:
+            z=head(r,s,ap,res,ac); p=z>=0; ref_m=frame_mask(sp,len(p)); c+=[(p&ref_m).sum(),(p&~ref_m).sum(),(~p&ref_m).sum()]
+        P=c[0]/(c[0]+c[1]); R=c[0]/(c[0]+c[2])
+        n23=np.mean([(head(r,s,ap,res,ac)>=0).mean() for s in sub_nz[:2]]); n40=np.mean([(head(r,s,ap,res,ac)>=0).mean() for s in sub_nz[2:]])
+        fps=[]
+        for s,g in sub_g:
+            z=head(r,s,ap,res,ac); t=(np.arange(len(z))+.5)*.08; fps.append((z[(t>=g['stretch'][0])&(t=0).mean())
+        print(f'  ({ap},{ac},res={res}) | F1 {100*2*P*R/(P+R):.1f} | {100*n23:.0f} | {100*n40:.0f} | {100*np.mean(fps):.0f}',flush=True)
diff --git a/scripts/vad_bench/noise_dive/a_faith.py b/scripts/vad_bench/noise_dive/a_faith.py
new file mode 100644
index 0000000..398303f
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/a_faith.py
@@ -0,0 +1,23 @@
+import os
+FUSION=os.environ.get('FUSION_DIR','../fusion')  # scripts/vad_bench/fusion with data/ built by prep_libri.py
+import sys,numpy as np,json
+from vadlib import *; from sig import *
+os.makedirs('wav',exist_ok=True)
+lib=list(np.load(FUSION+'/data/libri.npy',allow_pickle=True)); r=np.random.default_rng(0)
+N=30*SR
+sp=np.concatenate(lib[10:16]).astype(np.float32)[:N]
+sp2=np.concatenate(lib[40:46]).astype(np.float32)[:N]
+spec=ltass(lib[:60])
+sigs={'speech_a':sp,'speech_b':sp2,'white':at_db(white(N,r),-25),'pink':at_db(colored(N,r,1),-25),'brown':at_db(colored(N,r,2),-25),
+ 'speechshaped':at_db(shaped(N,r,spec),-25),'silence':np.zeros(N,np.float32),'dither_1e-4':(r.standard_normal(N)*1e-4).astype(np.float32)}
+rows=[]
+for k,x in sigs.items():
+    wr(f'wav/{k}.wav',x)
+    for m,g in [('redux','redux'),('ultra','ultra'),('ultra','ultra-f16')]:
+        p=ref(m).probs(sf.read(f'wav/{k}.wav',dtype='float32')[0]); c=cli_probs(g,f'wav/{k}.wav'); n=min(len(p),len(c)); e=np.abs(p[:n]-c[:n])
+        rows.append(dict(sig=k,model=g,n=n,maxdiff=float(e.max()),meandiff=float(e.mean()),ref_speech_frac=float((p>=.5).mean()),cli_speech_frac=float((c>=.5).mean()),ref_mean_p=float(p.mean())))
+        pass
+json.dump(rows,open('res_A.json','w'),indent=1)
+
+print('| signal | model file | frames | max abs diff | mean abs diff | speech frac ref | speech frac CLI |\n|---|---|---|---|---|---|---|')
+for r in rows: print(f"| {r['sig']} | {r['model']} | {r['n']} | {r['maxdiff']:.2e} | {r['meandiff']:.2e} | {100*r['ref_speech_frac']:.1f}% | {100*r['cli_speech_frac']:.1f}% |")
diff --git a/scripts/vad_bench/noise_dive/b10_DS.py b/scripts/vad_bench/noise_dive/b10_DS.py
new file mode 100644
index 0000000..f66e547
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/b10_DS.py
@@ -0,0 +1,34 @@
+"""Two-variable predictor of the head's response on a noise stretch: D (mean normalised level of the stretch) and S (temporal std of the
+normalised log-mel inside the stretch, mean over bins = fluctuation scale s of the feature-space probe). Pool talk+inserted-noise clips (b8) and the fusion gap clips."""
+import numpy as np,soundfile as sf,json
+from vadlib import *; from sig import *; import corpus
+from sklearn.linear_model import LinearRegression
+import matplotlib; matplotlib.use('Agg'); import matplotlib.pyplot as plt
+pts=[]
+def add(y,a,b,src,kind):
+    mn=norm_mel(raw_logmel(y)).numpy(); seg=mn[int((a+1)*100):int((b-1)*100)]; D=float(seg.mean()); S=float(seg.std(0).mean())
+    for m in ['redux','ultra']:
+        z=ref(m).logits(y); t=(np.arange(len(z))+.5)*.08; st=(t>=a+1)&(t=0).mean()),med=float(np.median(z[st]))))
+for g in corpus.gap_clips():
+    if g['k']<6: add(g['y'],g['stretch'][0],g['stretch'][1],'gap',g['kind'])
+x=sf.read('talk/JaneMcGonigal-merged.wav',dtype='float32')[0][60*SR:300*SR]; rdb=20*np.log10(rms(x)); r=np.random.default_rng(12)
+for kind in ['white','pink']:
+    for rel in [-35,-25,-15,-5]:
+        for L in [10,30,60]:
+            s0=int(r.integers(0,100))*SR; sp=120-L; a=sp//2
+            n=at_db(white(L*SR,r) if kind=='white' else colored(L*SR,r,1),rdb+rel)
+            y=np.concatenate([x[s0:s0+a*SR],n,x[s0+a*SR:s0+sp*SR]]).astype(np.float32); add(y,a,a+L,'talk+insert',kind)
+json.dump(pts,open('res_B10.json','w'))
+for m in ['redux','ultra']:
+    P=[p for p in pts if p['m']==m]; X=np.array([[p['D'],p['S']] for p in P]); M=np.clip(np.array([p['med'] for p in P]),-10,10)
+    for nm,cols in [('D only',[0]),('S only',[1]),('D and S',[0,1])]:
+        lr=LinearRegression().fit(X[:,cols],M); print(m,nm,'R2 of median logit: %.2f'%lr.score(X[:,cols],M),'coef',lr.coef_.round(2))
+    for src in ['gap','talk+insert']:
+        Q=[p for p in P if p['src']==src]; print(f'   {src:12s} n={len(Q)} mean D {np.mean([p["D"] for p in Q]):+.2f} mean S {np.mean([p["S"] for p in Q]):.2f} FA {100*np.mean([p["fa"] for p in Q]):.0f}% median logit {np.mean([p["med"] for p in Q]):+.2f}')
+fig,axs=plt.subplots(1,2,figsize=(12,4.2),sharey=True)
+for ax,m in zip(axs,['redux','ultra']):
+    for src,c in [('gap','C3'),('talk+insert','C0')]:
+        Q=[p for p in pts if p['m']==m and p['src']==src]; sc=ax.scatter([p['D'] for p in Q],[p['S'] for p in Q],c=[p['med'] for p in Q],cmap='coolwarm',vmin=-6,vmax=3,marker='o' if src=='gap' else 's',s=30,edgecolors='k',linewidths=.4,label=src)
+    ax.set_xlabel('D: mean normalised level of the stretch'); ax.set_title(f'{m}: colour = median logit in the stretch'); ax.legend(fontsize=8)
+axs[0].set_ylabel('S: temporal std of normalised features in the stretch'); plt.colorbar(sc,ax=axs.ravel().tolist()); plt.savefig('fig_D_S.png',dpi=110,bbox_inches='tight')
diff --git a/scripts/vad_bench/noise_dive/b1_amp.py b/scripts/vad_bench/noise_dive/b1_amp.py
new file mode 100644
index 0000000..1669ea8
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/b1_amp.py
@@ -0,0 +1,35 @@
+import os
+FUSION=os.environ.get('FUSION_DIR','../fusion')  # scripts/vad_bench/fusion with data/ built by prep_libri.py
+import numpy as np,json
+from vadlib import *; from sig import *
+N=30*SR; r=np.random.default_rng(1)
+lib=list(np.load(FUSION+'/data/libri.npy',allow_pickle=True)); spec=ltass(lib[:60])
+base={'white':white(N,r),'pink':colored(N,r,1),'brown':colored(N,r,2),'speechshaped':shaped(N,r,spec)}
+sp=np.concatenate(lib[10:16]).astype(np.float32)[:N]
+base['speech']=sp
+levels=list(range(-110,1,10))  # dBFS rms
+res={}
+print('| signal | '+' | '.join(f'{l}' for l in levels)+' |')
+for k,b in base.items():
+    row=[]
+    for l in levels:
+        x=at_db(b,l); x=np.clip(x,-1,1)   # float samples; clip like a wav would
+        row.append({m:dict(sp=float((ref(m).probs(x)>=.5).mean()),mp=float(ref(m).probs(x).mean())) for m in ['redux','ultra']})
+    res[k]=row
+    for m in ['redux','ultra']:
+        print(f'| {k} {m} | '+' | '.join(f'{100*c[m]["sp"]:.0f}' for c in row)+' |',flush=True)
+# exact invariance test: pure gain on top of fixed noise (float arithmetic, no clipping), -80..0 dB, and x1000
+b=at_db(base['white'],-20)
+print('\nPure gain on white@-20dBFS (speech-frame %, max |logit change| vs 0 dB gain)')
+ref0={m:ref(m).logits(b) for m in['redux','ultra']}
+g={}
+for gdb in [-80,-70,-60,-50,-40,-30,-20,-10,0]:
+    x=(b*10**(gdb/20)).astype(np.float32)
+    for m in ['redux','ultra']:
+        z=ref(m).logits(x); g[(gdb,m)]=(float((z>=0).mean()),float(np.abs(z-ref0[m]).max()),float(np.abs(z-ref0[m]).mean()))
+    print(gdb,{m:tuple(round(v,3) for v in g[(gdb,m)]) for m in['redux','ultra']})
+x=(at_db(base['white'],-80)*1000).astype(np.float32)   # 1000x
+b2=at_db(base['white'],-80)
+for m in ['redux','ultra']:
+    z=ref(m).logits(x); z0=ref(m).logits(b2); print('x1000 of -80dBFS noise',m,'speech%',100*(z>=0).mean(),'(base %',100*(z0>=0).mean(),') max|dz| to base',np.abs(z-z0).max())
+json.dump(res,open('res_B1.json','w'))
diff --git a/scripts/vad_bench/noise_dive/b2_signals.py b/scripts/vad_bench/noise_dive/b2_signals.py
new file mode 100644
index 0000000..95fbba2
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/b2_signals.py
@@ -0,0 +1,31 @@
+import os
+FUSION=os.environ.get('FUSION_DIR','../fusion')  # scripts/vad_bench/fusion with data/ built by prep_libri.py
+import numpy as np,json
+from vadlib import *; from sig import *
+N=30*SR; r=np.random.default_rng(3)
+lib=list(np.load(FUSION+'/data/libri.npy',allow_pickle=True))
+S={}
+S['digital silence']=np.zeros(N,np.float32)
+S['DC offset 0.1']=np.full(N,0.1,np.float32)
+for f in [100,440,1000,3000]: S[f'sine {f} Hz -20 dBFS']=sine(N,f)
+S['sine 1 kHz -60 dBFS']=sine(N,1000,-60)
+S['sine sweep 50-7500 Hz (30 s)']=sweep(N)
+S['sine sweep -50 dBFS']=sweep(N,db=-50)
+S['clicks 3/s -12 dBFS']=clicks(N,r,3,-12)
+S['clicks 10/s -12 dBFS']=clicks(N,r,10,-12)
+S['clicks 1/s -12 dBFS']=clicks(N,r,1,-12)
+S['hum 50 Hz+harmonics -25 dBFS']=hum(N,50,-25)
+S['hum 60 Hz+harmonics -25 dBFS']=hum(N,60,-25)
+S['hum 50 Hz + white -60 dBFS']=hum(N,50,-25)+at_db(white(N,r),-60)
+S['music-like chords -20 dBFS']=music(N,r)
+S['white -25 dBFS']=at_db(white(N,r),-25); S['pink -25 dBFS']=at_db(colored(N,r,1),-25); S['brown -25 dBFS']=at_db(colored(N,r,2),-25)
+S['white -70 dBFS']=at_db(white(N,r),-70)
+S['speech (LibriSpeech x6)']=np.concatenate(lib[10:16]).astype(np.float32)[:N]
+rows=[]; Z={}
+print('| signal | Redux speech frames % | Redux mean p | Ultra speech frames % | Ultra mean p |\n|---|---|---|---|---|')
+for k,x in S.items():
+    cell=[]
+    for m in ['redux','ultra']:
+        z=ref(m).logits(x.astype(np.float32)); Z[f'{k}|{m}']=z; p=sig(z); cell+= [f'{100*(p>=.5).mean():.1f}',f'{p.mean():.2f}']
+    print(f'| {k} | '+' | '.join(cell)+' |',flush=True)
+np.savez('res_B2_logits.npz',**{k.replace('|','@'):v for k,v in Z.items()})
diff --git a/scripts/vad_bench/noise_dive/b3_norm.py b/scripts/vad_bench/noise_dive/b3_norm.py
new file mode 100644
index 0000000..20ea098
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/b3_norm.py
@@ -0,0 +1,56 @@
+import numpy as np,json,torch
+from vadlib import *; from sig import *; import corpus
+S=np.load('fixed_stats.npz'); T=lambda a: torch.tensor(a)
+variants={'file':(None,None),'fixedA':(T(S['aM']),T(S['aS'])),'fixedB':(T(S['bM']),T(S['bS']))}
+models=['redux','ultra']
+cl=corpus.clean_clips(); gaps=corpus.gap_clips()
+R={}
+def f1(p,r):
+    n=min(len(p),len(r)); p=p[:n]; r=r[:n]; tp=(p&r).sum(); fp=(p&~r).sum(); fn=(~p&r).sum()
+    P=tp/max(1,tp+fp); Rc=tp/max(1,tp+fn); return P,Rc,2*P*Rc/max(1e-9,P+Rc)
+# 1. clean speech clips
+for v,(mu,sd) in variants.items():
+    for m in models:
+        c=np.zeros(3)
+        for y,spans in cl:
+            z=logits_blocks(m,y,mu,sd) if mu is None else ref(m).logits(y,mu,sd)
+            p=z>=0; r=frame_mask(spans,len(p)); n=len(p)
+            c+=[(p&r).sum(),(p&~r).sum(),(~p&r).sum()]
+        P=c[0]/(c[0]+c[1]); Rc=c[0]/(c[0]+c[2]); R[f'clean|{v}|{m}']=dict(P=P,R=Rc,F1=2*P*Rc/(P+Rc))
+        print('clean',v,m,R[f'clean|{v}|{m}'],flush=True)
+# 2. noise-only 30 s at absolute levels
+rng=np.random.default_rng(5); N=30*SR
+for kind in ['white','pink']:
+    for lvl in [-60,-50,-40,-30,-23,-15]:
+        fr={v:{m:[] for m in models} for v in variants}
+        for k in range(6):
+            b=at_db(white(N,rng) if kind=='white' else colored(N,rng,1),lvl)
+            for v,(mu,sd) in variants.items():
+                for m in models: fr[v][m].append((ref(m).logits(b,mu,sd)>=0).mean())
+        for v in variants:
+            for m in models: R[f'nz|{kind}|{lvl}|{v}|{m}']=float(np.mean(fr[v][m]))
+        print('noise-only',kind,lvl,{v:{m:round(100*np.mean(fr[v][m]),1) for m in models} for v in variants},flush=True)
+# 3. embedded stretch
+agg={}
+for g in gaps:
+    y=g['y']; ref_m=None
+    for v,(mu,sd) in variants.items():
+        for m in models:
+            z=logits_blocks(m,y,mu,sd) if mu is None else ref(m).logits(y,mu,sd); p=z>=0
+            n=len(p); t=(np.arange(n)+.5)*.08; st=(t>=g['stretch'][0])&(t=0).mean()),r2_lin6=r2te,r2_gbm6=gbr2,r2_spec128=rdr2,r2_energy_only=e1,corr=corr,coef=lr.coef_.tolist(),logit_mean=float(Z.mean()),logit_std=float(Z.std()))
+        print(f'{m:6s} {kn:22s} speech {100*(Z>=0).mean():5.1f}%  logit mean {Z.mean():+.2f} sd {Z.std():.2f} | held-out R2: 6-feature linear {r2te:+.2f}, 6-feature GBM {gbr2:+.2f}, energy only {e1:+.2f}, 128-bin window-mel ridge {rdr2:+.2f} | corr '+' '.join(f'{c:+.2f}' for c in corr)+f' | dominant {dom}',flush=True)
+print('features:',names)
+json.dump(res,open('res_B4.json','w'),indent=1)
diff --git a/scripts/vad_bench/noise_dive/b4b_probe.py b/scripts/vad_bench/noise_dive/b4b_probe.py
new file mode 100644
index 0000000..e0e16df
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/b4b_probe.py
@@ -0,0 +1,36 @@
+import os
+FUSION=os.environ.get('FUSION_DIR','../fusion')  # scripts/vad_bench/fusion with data/ built by prep_libri.py
+"""Feature-space probe: feed the head (via the real subsampler) normalised-mel inputs that we construct directly."""
+import numpy as np,json
+from vadlib import *; from sig import *
+import matplotlib; matplotlib.use('Agg'); import matplotlib.pyplot as plt
+rng=np.random.default_rng(8); N=30*SR; T=3000
+lib=list(np.load(FUSION+'/data/libri.npy',allow_pickle=True))
+sp=np.concatenate(lib[10:16]).astype(np.float32)[:N]
+res={}
+mel_sp=norm_mel(raw_logmel(sp))                      # speech as the net sees it
+mel_wn=norm_mel(raw_logmel(at_db(white(N,rng),-25)))
+iid=torch.tensor(rng.standard_normal((T,128)).astype(np.float32))   # iid N(0,1) features, no structure at all
+print('A) iid N(0,1) normalised features (no structure) -> speech frames %')
+for m in ['redux','ultra']:
+    r=ref(m); print(' ',m,100*(r.head_logits(r.sub_out(iid))>=0).float().mean().item(),'% ; mean logit',r.head_logits(r.sub_out(iid)).mean().item(), '; all-zero features logit',r.head_logits(r.sub_out(torch.zeros(T,128))).mean().item())
+print('B) offset d (in per-file std units) added to every bin and frame; scale s on the fluctuation (white noise normalised mel)')
+ds=np.arange(-3,3.01,0.5); ss=[0.25,0.5,1.0,1.5,2.0]
+fig,ax=plt.subplots(1,2,figsize=(11,4))
+for mi,m in enumerate(['redux','ultra']):
+    r=ref(m); Mi=np.zeros((len(ss),len(ds)))
+    for i,s in enumerate(ss):
+        for j,d in enumerate(ds):
+            x=mel_wn*s+d; Mi[i,j]=(r.head_logits(r.sub_out(x))>=0).float().mean().item()
+    res[f'offset_scale|{m}']=Mi.tolist()
+    im=ax[mi].imshow(100*Mi,origin='lower',aspect='auto',extent=[ds[0]-.25,ds[-1]+.25,-0.5,len(ss)-.5],vmin=0,vmax=100,cmap='viridis')
+    ax[mi].set_yticks(range(len(ss))); ax[mi].set_yticklabels(ss); ax[mi].set_xlabel('offset d (std units of the file)'); ax[mi].set_ylabel('fluctuation scale s'); ax[mi].set_title(f'{m}: % frames speech, white-noise features*s+d')
+    plt.colorbar(im,ax=ax[mi])
+    print(m); print('   s\\d '+' '.join(f'{d:5.1f}' for d in ds))
+    for i,s in enumerate(ss): print(f'  {s:4.2f} '+' '.join(f'{100*v:5.0f}' for v in Mi[i]))
+plt.tight_layout(); plt.savefig('fig_offset_scale.png',dpi=110); plt.close()
+# C) speech file: shift the whole normalised mel by d
+print('C) real speech normalised mel shifted by d: speech frames % (of frames)')
+for m in ['redux','ultra']:
+    r=ref(m); print(' ',m,[ (d,round(100*(r.head_logits(r.sub_out(mel_sp+d))>=0).float().mean().item())) for d in [-2,-1,-0.5,0,0.5,1,2]])
+json.dump(res,open('res_B4b.json','w'))
diff --git a/scripts/vad_bench/noise_dive/b5_logits.py b/scripts/vad_bench/noise_dive/b5_logits.py
new file mode 100644
index 0000000..dc8b02b
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/b5_logits.py
@@ -0,0 +1,53 @@
+import numpy as np,json
+from vadlib import *; from sig import *; import corpus
+from sklearn.metrics import roc_auc_score, roc_curve
+import matplotlib; matplotlib.use('Agg'); import matplotlib.pyplot as plt
+models=['redux','ultra']; cl=corpus.clean_clips(); gaps=corpus.gap_clips()
+rng=np.random.default_rng(5); N=30*SR
+D={m:dict(speech_clean=[],pause_clean=[],speech_noisy={},stretch={},nz={}) for m in models}
+for m in models:
+    for y,spans in cl:
+        z=logits_blocks(m,y); sp=frame_mask(spans,len(z))
+        # pause frames: >=0.3 s from any speech (gap between utterances, digital -60 dBFS dither)
+        t=(np.arange(len(z))+.5)*.08; far=np.ones(len(z),bool)
+        for a,b in spans: far&=~((t>a-0.3)&(t=g['stretch'][0])&(t=0).mean():.0f}% / {100*(b>=0).mean():.0f}% |')
+        out[f'{m}|{nm}']=dict(auc=auc,thr=float(thr),tpr=tpr,fpr=fpr,thrF1=float(thF))
+json.dump(out,open('res_B5.json','w'),indent=1)
+# logit percentiles
+print('\nlogit percentiles (5,25,50,75,95):')
+for m in models:
+    d=D[m]
+    for nm,z in [('speech clean',cat(d['speech_clean'])),('pause (digital -60 dBFS dither) in clean clips',cat(d['pause_clean'])),('noise-only -23 dBFS',d['nz'][-23]),('noise-only -40 dBFS',d['nz'][-40]),('noise-only -50 dBFS',d['nz'][-50]),('embedded stretch 20 dB',cat(d['stretch'][20])),('embedded stretch 5 dB',cat(d['stretch'][5])),('embedded stretch 0 dB',cat(d['stretch'][0]))]:
+        print(f'{m:6s} {nm:48s}',np.percentile(z,[5,25,50,75,95]).round(2))
+fig,axs=plt.subplots(1,2,figsize=(12,4),sharey=True)
+for ax,m in zip(axs,models):
+    d=D[m]; bins=np.linspace(-20,25,90)
+    for nm,z,c in [('speech (clean clips)',cat(d['speech_clean']),'C0'),('pauses in clean clips',cat(d['pause_clean']),'C1'),('noise-only -23 dBFS',d['nz'][-23],'C3'),('noise-only -50 dBFS',d['nz'][-50],'C2'),('embedded stretch 5 dB',cat(d['stretch'][5]),'C4')]:
+        ax.hist(np.clip(z,-20,25),bins=bins,density=True,histtype='step',label=nm,color=c,lw=1.6)
+    ax.axvline(0,color='k',ls=':'); ax.set_title(m+': pre-sigmoid logit (0 = default threshold 0.5)'); ax.set_xlabel('logit'); ax.legend(fontsize=8)
+plt.tight_layout(); plt.savefig('fig_logit_hist.png',dpi=110)
diff --git a/scripts/vad_bench/noise_dive/b6_types.py b/scripts/vad_bench/noise_dive/b6_types.py
new file mode 100644
index 0000000..1200a03
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/b6_types.py
@@ -0,0 +1,41 @@
+import numpy as np,json
+from vadlib import *; from sig import *; import corpus
+import matplotlib; matplotlib.use('Agg'); import matplotlib.pyplot as plt
+models=['redux','ultra']; fold_of=corpus.folds(); libri=corpus.libri; spk=corpus.spk
+spec=ltass(libri[:60])
+def mk(k, noise_fn, snr, whole):
+    f=k%4; idx=[i for i in range(len(libri)) if fold_of[spk[i]]==f]; r=np.random.default_rng(900+k); idx=list(r.permutation(idx))[:5]
+    us=[libri[i] for i in idx]
+    y1,s1=corpus.FL.build(us[:2],np.random.default_rng(400+k)); y2,s2=corpus.FL.build(us[2:],np.random.default_rng(500+k))
+    off=len(y1)/16000+30.0
+    y=np.concatenate([y1,np.random.default_rng(600+k).standard_normal(30*16000)*1e-3,y2]).astype(np.float32)
+    spans=s1+[(a+off,b+off) for a,b in s2]
+    P=np.mean(np.concatenate([y[int(a*SR):int(b*SR)] for a,b in spans])**2)
+    st=(s1[-1][1]+0.5,s2[0][0]+off-0.5); a,b=int(st[0]*SR),int(st[1]*SR)
+    nr=np.random.default_rng(77+k)
+    if whole:
+        n=noise_fn(len(y),nr); n=n/ (rms(n)+1e-12)*np.sqrt(P/10**(snr/10)); y=y+n.astype(np.float32)
+    else:
+        n=noise_fn(b-a,nr); n=n/(rms(n)+1e-12)*np.sqrt(P/10**(snr/10)); y=y.copy(); y[a:b]+=n.astype(np.float32)
+    return np.clip(y,-1,1).astype(np.float32),st
+types={'white':(lambda n,r:white(n,r),True),'pink':(lambda n,r:colored(n,r,1),True),'brown':(lambda n,r:colored(n,r,2),True),'speech-shaped':(lambda n,r:shaped(n,r,spec),True),
+ 'hum 50 Hz':(lambda n,r:hum(n,50,-10),False),'sine 1 kHz':(lambda n,r:sine(n,1000,-10),False),'clicks 3/s':(lambda n,r:clicks(n,r,3,-10),False),'clicks 10/s':(lambda n,r:clicks(n,r,10,-10),False),'music-like':(lambda n,r:music(n,r,-10),False)}
+snrs=[-10,0,10,20,30,40]
+R={}
+print('Embedded-stretch false-alarm frame % at threshold 0.5 ; rows=type, cols=SNR re speech (dB) ; 6 clips per cell')
+for m in models:
+    print(f'\n{m}:'); print('| type | '+' | '.join(f'{s} dB' for s in snrs)+' |'); print('|---|'+'---|'*len(snrs))
+    for tn,(fn,whole) in types.items():
+        row=[]
+        for s in snrs:
+            fps=[];mz=[]
+            for k in range(6):
+                y,st=mk(k,fn,s,whole); z=ref(m).logits(y); t=(np.arange(len(z))+.5)*.08; msk=(t>=st[0])&(t=0).mean()); mz.append(np.median(z[msk]))
+            R[f'{m}|{tn}|{s}']=(float(np.mean(fps)),float(np.mean(mz))); row.append(f'{100*np.mean(fps):.0f}')
+        print(f'| {tn} | '+' | '.join(row)+' |',flush=True)
+json.dump(R,open('res_B6.json','w'))
+fig,axs=plt.subplots(1,2,figsize=(12,4),sharey=True)
+for ax,m in zip(axs,models):
+    for tn in types: ax.plot(snrs,[100*R[f'{m}|{tn}|{s}'][0] for s in snrs],marker='o',label=tn,ls='-' if types[tn][1] else '--')
+    ax.set_title(f'{m}: stretch false alarms vs level of the stretch below speech'); ax.set_xlabel('stretch level re speech power (dB below speech)'); ax.set_ylabel('% stretch frames called speech'); ax.legend(fontsize=7)
+plt.tight_layout(); plt.savefig('fig_types_vs_level.png',dpi=110)
diff --git a/scripts/vad_bench/noise_dive/b7_fraction.py b/scripts/vad_bench/noise_dive/b7_fraction.py
new file mode 100644
index 0000000..303d7fc
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/b7_fraction.py
@@ -0,0 +1,31 @@
+"""Mechanism: share of the normalisation window that is noise. 120 s clip = (120-L) s of a real talk + L s of noise in the middle."""
+import numpy as np,soundfile as sf,json
+from vadlib import *; from sig import *
+import matplotlib; matplotlib.use('Agg'); import matplotlib.pyplot as plt
+x=sf.read('talk/JaneMcGonigal-merged.wav',dtype='float32')[0][60*SR:300*SR]
+rdb=20*np.log10(rms(x)); r=np.random.default_rng(2)
+Ls=[5,10,20,30,45,60,90]; rels=[-35,-25,-15,-5]; R={}
+fig,axs=plt.subplots(2,2,figsize=(11,7),sharey=True)
+for mi,m in enumerate(['redux','ultra']):
+    for ki,kind in enumerate(['white','pink']):
+        ax=axs[mi][ki]
+        for rel in rels:
+            ys=[]
+            for L in Ls:
+                fp=[]
+                for rep in range(3):
+                    s0=int(r.integers(0,100))*SR; sp=120-L; a=sp//2
+                    pre=x[s0:s0+a*SR]; post=x[s0+a*SR:s0+sp*SR]
+                    n=at_db(white(L*SR,r) if kind=='white' else colored(L*SR,r,1),rdb+rel)
+                    y=np.concatenate([pre,n,post]).astype(np.float32)
+                    z=ref(m).logits(y); t=(np.arange(len(z))+.5)*.08; st=(t>=a+1)&(t=0).mean())
+                ys.append(100*np.mean(fp)); R[f'{m}|{kind}|{rel}|{L}']=ys[-1]
+            ax.plot([100*L/120 for L in Ls],ys,marker='o',label=f'noise {rel} dB re talk RMS')
+        ax.set_title(f'{m}, {kind} noise stretch'); ax.set_xlabel('noise share of the 120 s normalisation window (%)'); ax.set_ylabel('% stretch frames called speech'); ax.grid(alpha=.3)
+axs[0][0].legend(fontsize=8); plt.tight_layout(); plt.savefig('fig_noise_fraction.png',dpi=110)
+print('talk rms dBFS %.1f'%rdb)
+for m in ['redux','ultra']:
+    for kind in ['white','pink']:
+        print(f'\n{m} {kind}: stretch FA % ; rows noise level re talk RMS, cols noise share of window (L s of 120)'); print('| level | '+' | '.join(f'{L} s ({100*L//120}%)' for L in Ls)+' |'); print('|---|'+'---|'*len(Ls))
+        for rel in rels: print(f'| {rel} dB | '+' | '.join(f'{R[f"{m}|{kind}|{rel}|{L}"]:.0f}' for L in Ls)+' |')
+json.dump(R,open('res_B7.json','w'))
diff --git a/scripts/vad_bench/noise_dive/b8_delta.py b/scripts/vad_bench/noise_dive/b8_delta.py
new file mode 100644
index 0000000..dfdbe3c
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/b8_delta.py
@@ -0,0 +1,31 @@
+"""Unifying predictor: normalised level of the stretch, D = mean over stretch frames and mel bins of the per-file-normalised log-mel
+(0 = the file's average level, -1 = one per-bin std below it). Relate D to the head's decision on the stretch."""
+import numpy as np,soundfile as sf,json
+from vadlib import *; from sig import *
+import matplotlib; matplotlib.use('Agg'); import matplotlib.pyplot as plt
+x=sf.read('talk/JaneMcGonigal-merged.wav',dtype='float32')[0][60*SR:300*SR]; rdb=20*np.log10(rms(x)); r=np.random.default_rng(12)
+pts=[]
+for kind in ['white','pink','speech-shaped','music']:
+    for rel in [-35,-30,-25,-20,-15,-10,-5]:
+        for L in [10,30,60]:
+            for rep in range(2):
+                s0=int(r.integers(0,100))*SR; sp=120-L; a=sp//2
+                n=at_db(white(L*SR,r) if kind=='white' else colored(L*SR,r,1) if kind=='pink' else shaped(L*SR,r,ltass([x[:SR*60]])) if kind=='speech-shaped' else music(L*SR,r,0),rdb+rel)
+                y=np.concatenate([x[s0:s0+a*SR],n,x[s0+a*SR:s0+sp*SR]]).astype(np.float32)
+                mn=norm_mel(raw_logmel(y)); fr0=int((a+1)*100); fr1=int((a+L-1)*100); D=float(mn[fr0:fr1].mean())
+                for m in ['redux','ultra']:
+                    z=ref(m).logits(y); t=(np.arange(len(z))+.5)*.08; st=(t>=a+1)&(t=0).mean()),med=float(np.median(z[st]))))
+json.dump(pts,open('res_B8.json','w'))
+fig,axs=plt.subplots(1,2,figsize=(12,4.2),sharey=True)
+for ax,m in zip(axs,['redux','ultra']):
+    for kind,c in zip(['white','pink','speech-shaped','music'],['C0','C1','C3','C2']):
+        P=[p for p in pts if p['m']==m and p['kind']==kind]; ax.scatter([p['D'] for p in P],[p['med'] for p in P],s=18,c=c,label=kind,alpha=.7)
+    ax.axhline(0,color='k',ls=':'); ax.set_xlabel('normalised level of the stretch D (std units; 0 = file average)'); ax.set_title(f'{m}: median logit in a noise stretch vs D'); ax.legend(fontsize=8); ax.grid(alpha=.3)
+axs[0].set_ylabel('median logit in stretch'); plt.tight_layout(); plt.savefig('fig_logit_vs_delta.png',dpi=110)
+for m in ['redux','ultra']:
+    P=[p for p in pts if p['m']==m]; D=np.array([p['D'] for p in P]); M=np.array([p['med'] for p in P]); F=np.array([p['fa'] for p in P])
+    print(m,'corr(D, median logit) %.2f ; corr(D, FA) %.2f'%(np.corrcoef(D,M)[0,1],np.corrcoef(D,F)[0,1]))
+    for lo,hi in [(-9,-2),(-2,-1.5),(-1.5,-1),(-1,-.5),(-.5,0),(0,.5),(.5,9)]:
+        s=(D>=lo)&(D=a)&(t=0).mean()),med=float(np.median(z[st]))))
+json.dump(rows,open('res_B9.json','w'))
+for m in ['redux','ultra']:
+    print(m)
+    for snr in [20,10,5,0]:
+        P=[r for r in rows if r['m']==m and r['snr']==snr]; print(f'  SNR {snr:2d}: mean D {np.mean([r["D"] for r in P]):+.2f}  FA {100*np.mean([r["fa"] for r in P]):.0f}%  median logit {np.mean([r["med"] for r in P]):+.2f}')
diff --git a/scripts/vad_bench/noise_dive/cli_check.py b/scripts/vad_bench/noise_dive/cli_check.py
new file mode 100644
index 0000000..7c0c63b
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/cli_check.py
@@ -0,0 +1,15 @@
+"""Cross-check with the shipped parakeet-cli on a few rebuilt clips: gap clips (embedded stretch) and amplitude."""
+import numpy as np,soundfile as sf,subprocess,json,os
+from vadlib import *; from sig import *; import corpus
+os.makedirs('wav',exist_ok=True); gaps=corpus.gap_clips(); res=[]
+for g in gaps:
+    if g['k'] not in (0,1) or g['snr'] not in (20,5,0) or g['kind']!='white': continue
+    wr(f"wav/{g['name']}.wav",g['y']); n=g['name']
+    for m,gg in [('redux','redux'),('ultra','ultra')]:
+        c=cli_probs(gg,f'wav/{n}.wav'); z=ref(m).logits(sf.read(f'wav/{n}.wav',dtype='float32')[0]); p=sig(z)
+        t=(np.arange(len(c))+.5)*.08; st=(t>=g['stretch'][0])&(t=.5).mean(),100*(p[:len(c)][st]>=.5).mean(),np.abs(c-p[:len(c)]).max()),flush=True)
+r=np.random.default_rng(1); b=white(30*SR,r)
+for lv in [-20,-50,-60,-70,-80]:
+    wr(f'wav/amp_{lv}.wav',at_db(b,lv))
+    print('amp',lv,{m:round(100*(cli_probs(m,f'wav/amp_{lv}.wav')>=.5).mean(),1) for m in ['redux','ultra']},flush=True)
diff --git a/scripts/vad_bench/noise_dive/corpus.py b/scripts/vad_bench/noise_dive/corpus.py
new file mode 100644
index 0000000..fc62b10
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/corpus.py
@@ -0,0 +1,36 @@
+import os
+FUSION=os.environ.get('FUSION_DIR','../fusion')  # scripts/vad_bench/fusion with data/ built by prep_libri.py
+"""Rebuild fusion/ clips in memory with the same generators and seeds (fusion/lib.py is imported read-only)."""
+import sys,json,numpy as np
+sys.path.insert(0,FUSION)
+import lib as FL
+libri=list(np.load(FUSION+"/data/libri.npy",allow_pickle=True)); spk=np.load(FUSION+"/data/spk.npy")
+def folds():
+    rng=np.random.default_rng(11); sp_ids=rng.permutation(sorted(set(spk))); return {s:i%4 for i,s in enumerate(sp_ids)}
+def speech_sets():
+    fold_of=folds(); rng=np.random.default_rng(11); rng.permutation(sorted(set(spk)))
+    sets=[]
+    for f in range(4):
+        idx=[i for i in range(len(libri)) if fold_of[spk[i]]==f]; idx=list(rng.permutation(idx))
+        for k in range(min(10,len(idx)//5)): sets.append((f,[libri[i] for i in idx[k*5:(k+1)*5]]))
+    return sets
+def clean_clips():
+    out=[]
+    for k,(f,us) in enumerate(speech_sets()):
+        y,spans=FL.build(us,np.random.default_rng(1000+k)); out.append((y,spans))
+    return out
+def gap_clips():
+    """12 per (white|pink) x (20,10,5,0 dB): 2 utts, 30 s noise stretch, 3 utts; noise over the whole clip at the SNR (speech power)."""
+    fold_of=folds(); meta=[]
+    for ci,(kind,snr) in enumerate([(k,s) for k in("white","pink") for s in(20,10,5,0)]):
+        for k in range(12):
+            f=k%4; idx=[i for i in range(len(libri)) if fold_of[spk[i]]==f]; r=np.random.default_rng(300+ci*50+k); idx=list(r.permutation(idx))[:5]
+            us=[libri[i] for i in idx]
+            y1,s1=FL.build(us[:2],np.random.default_rng(400+k)); y2,s2=FL.build(us[2:],np.random.default_rng(500+k))
+            off=len(y1)/16000+30.0
+            y=np.concatenate([y1,np.random.default_rng(600+k).standard_normal(30*16000)*1e-3,y2]).astype(np.float32)
+            spans=s1+[(a+off,b+off) for a,b in s2]
+            clean=y.copy()
+            y=FL.add_noise(y,spans,kind,snr,np.random.default_rng(7000+ci*100+k))
+            meta.append(dict(name=f"gap-{kind}{snr}_{k:02d}",kind=kind,snr=snr,k=k,y=y,clean=clean,spans=spans,stretch=(s1[-1][1]+0.5,s2[0][0]+off-0.5)))
+    return meta
diff --git a/scripts/vad_bench/noise_dive/d_analyze.py b/scripts/vad_bench/noise_dive/d_analyze.py
new file mode 100644
index 0000000..c1fec1f
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/d_analyze.py
@@ -0,0 +1,26 @@
+import json,os
+meta=json.load(open('talk/d/meta.json')); a,b=meta['ins']
+def load(p):
+    try: return json.load(open(p))
+    except Exception: return None
+def words_in(d,lo,hi): return [w for w in d['words'] if lo<=w['start']a and s['start']0: print(r['model'],r['file'],'| novad:',r['sample0'][:90],'| vad:',r['sample1'][:90])
diff --git a/scripts/vad_bench/noise_dive/d_mit.py b/scripts/vad_bench/noise_dive/d_mit.py
new file mode 100644
index 0000000..90669dd
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/d_mit.py
@@ -0,0 +1,19 @@
+import os
+FUSION=os.environ.get('FUSION_DIR','../fusion')  # scripts/vad_bench/fusion with data/ built by prep_libri.py
+import numpy as np,json,soundfile as sf,sys
+sys.path.insert(0,FUSION); import fl
+from vadlib import *
+meta=json.load(open('talk/d/meta.json')); a,b=meta['ins']
+def runconf(z,q):
+    keep=np.zeros(len(z),bool)
+    for i,j in zip(*fl.runs(z>=0)):
+        if np.median(z[i:j])>=q: keep[i:j]=True
+    return keep
+print('insert (60 s) frames called speech %, by mitigation (block-normalised logits, shipped segmenter smoothing not applied)\n| file | model | thr 0.5 | thr 0.7 | thr 0.9 | runconf 2.5 | talk speech frames kept (outside insert, vs thr 0.5) runconf 2.5 |\n|---|---|---|---|---|---|---|')
+for f,tn,rel in meta['files']:
+    if f=='base': continue
+    x=sf.read(f'talk/d/{f}.wav',dtype='float32')[0]
+    for m in ['redux','ultra']:
+        z=logits_blocks(m,x); t=(np.arange(len(z))+.5)*.08; st=(t>=a+1)&(tb+1)
+        rc=runconf(z,2.5); base=z>=0
+        print(f'| {f} | {m} | {100*base[st].mean():.0f} | {100*(z[st]>=np.log(.7/.3)).mean():.0f} | {100*(z[st]>=np.log(.9/.1)).mean():.0f} | {100*rc[st].mean():.0f} | {100*rc[out].sum()/base[out].sum():.1f}% |',flush=True)
diff --git a/scripts/vad_bench/noise_dive/d_prep.py b/scripts/vad_bench/noise_dive/d_prep.py
new file mode 100644
index 0000000..97e6914
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/d_prep.py
@@ -0,0 +1,21 @@
+import numpy as np,soundfile as sf,json
+from sig import *
+x,sr=sf.read('talk/JaneMcGonigal-merged.wav',dtype='float32'); assert sr==16000
+ex=x[60*SR:300*SR]                                  # 4 min excerpt
+# find the quietest 1 s window between 100 and 140 s of the excerpt to cut at (a natural pause)
+fr=len(ex)//1600; e=10*np.log10((ex[:fr*1600].reshape(fr,1600)**2).mean(1)+1e-12)
+c=[(e[i:i+10].mean(),i) for i in range(1000,1400)]; _,i0=min(c); cut=(i0+5)*1600     # centre of that window
+speech_rms_db=20*np.log10(rms(ex)); print('excerpt rms dBFS',speech_rms_db,'cut at',cut/SR,'s')
+r=np.random.default_rng(4); L=60*SR
+gens={'white':lambda: white(L,r),'pink':lambda: colored(L,r,1),'clicks':lambda: clicks(L,r,3,0),'music':lambda: music(L,r,0)}
+meta={'cut':cut/SR,'ins':[cut/SR,cut/SR+60],'rms_db':speech_rms_db,'files':[]}
+sf.write('talk/d/base.wav',ex,16000,subtype='PCM_16'); meta['files'].append(('base',None,None))
+for tn,g in gens.items():
+    n0=g()
+    for rel in [-35,-20,-5]:
+        n=at_db(n0,speech_rms_db+rel) if tn!='clicks' else (n0/ (np.abs(n0).max()+1e-9) * 10**((speech_rms_db+rel+14)/20)).astype(np.float32)  # clicks: set RMS ~ rel via crest ~ calibrate below
+        if tn=='clicks': n=at_db(n0,speech_rms_db+rel)
+        y=np.concatenate([ex[:cut],np.clip(n,-1,1),ex[cut:]]).astype(np.float32)
+        nm=f'{tn}_{rel}'; sf.write(f'talk/d/{nm}.wav',y,16000,subtype='PCM_16'); meta['files'].append((nm,tn,rel))
+json.dump(meta,open('talk/d/meta.json','w'))
+print(len(meta['files']),'files')
diff --git a/scripts/vad_bench/noise_dive/d_probs.py b/scripts/vad_bench/noise_dive/d_probs.py
new file mode 100644
index 0000000..3025c77
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/d_probs.py
@@ -0,0 +1,14 @@
+import numpy as np,json,torch,soundfile as sf
+from vadlib import *
+S=np.load('fixed_stats.npz'); fa=(torch.tensor(S['aM']),torch.tensor(S['aS']))
+meta=json.load(open('talk/d/meta.json')); a,b=meta['ins']; res={}
+print('| file | model | insert frames called speech %: block-120 s norm (shipped) | whole-file norm | fixed norm A | median logit (shipped) |\n|---|---|---|---|---|---|')
+for f,tn,rel in meta['files']:
+    x=sf.read(f'talk/d/{f}.wav',dtype='float32')[0]
+    for m in ['redux','ultra']:
+        zb=logits_blocks(m,x); zf=ref(m).logits(x); zx=ref(m).logits(x,*fa)
+        t=(np.arange(len(zb))+.5)*.08; st=(t>=a+1)&(t=a-30)&(t=0).mean()),whole=float((zf[:len(zb)][st]>=0).mean()),fixed=float((zx[:len(zb)][st]>=0).mean()),med=float(np.median(zb[st])))
+        r=res[f'{f}|{m}']; print(f'| {f} | {m} | {100*r["block"]:.0f} | {100*r["whole"]:.0f} | {100*r["fixed"]:.0f} | {r["med"]:+.1f} |',flush=True)
+json.dump(res,open('res_D_probs.json','w'))
diff --git a/scripts/vad_bench/noise_dive/d_run.py b/scripts/vad_bench/noise_dive/d_run.py
new file mode 100644
index 0000000..7a39b7b
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/d_run.py
@@ -0,0 +1,17 @@
+import json,subprocess,sys,os,time
+CLI=os.environ.get('PARAKEET_CLI','build/examples/cli/parakeet-cli'); S=os.environ.get('GGUF_ROOT','..')
+M={'ultra':S+'/gguf/ultra-q8_0.gguf','redux':S+'/gguf/redux-keep.gguf'}
+meta=json.load(open('talk/d/meta.json')); os.makedirs('talk/out',exist_ok=True)
+files=[f[0] for f in meta['files']]
+for m in ['ultra','redux']:
+    for f in files:
+        wav=f'talk/d/{f}.wav'
+        o=f'talk/out/{f}.{m}.segments.json'
+        if not os.path.exists(o):
+            r=subprocess.run([CLI,'vad','--model',M[m],'--input',wav,'--mode','segments','--threads','4'],capture_output=True,text=True); open(o,'w').write(r.stdout)
+        for vad in [1,0]:
+            o=f'talk/out/{f}.{m}.vad{vad}.json'
+            if os.path.exists(o): continue
+            t=time.time()
+            r=subprocess.run([CLI,'transcribe','--model',M[m],'--input',wav,'--json','--timestamps','--threads','4']+(['--vad'] if vad else []),capture_output=True,text=True)
+            open(o,'w').write(r.stdout); print(m,f,vad,round(time.time()-t,1),r.returncode,len(r.stdout),flush=True)
diff --git a/scripts/vad_bench/noise_dive/e2_talk.py b/scripts/vad_bench/noise_dive/e2_talk.py
new file mode 100644
index 0000000..7f1f468
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/e2_talk.py
@@ -0,0 +1,24 @@
+import os
+FUSION=os.environ.get('FUSION_DIR','../fusion')  # scripts/vad_bench/fusion with data/ built by prep_libri.py
+import numpy as np,soundfile as sf,sys,onnxruntime as ort,os
+sys.path.insert(0,FUSION); import fl
+from vadlib import *
+x=sf.read('talk/JaneMcGonigal-merged.wav',dtype='float32')[0]
+so=ort.SessionOptions(); so.intra_op_num_threads=2
+sess=ort.InferenceSession(os.environ['SILERO_ONNX'],so,providers=["CPUExecutionProvider"])
+y=np.pad(x,(0,(-len(x))%512)); st=np.zeros((2,1,128),np.float32); ctx=np.zeros(64,np.float32); sil=[]
+for i in range(len(y)//512):
+    xx=np.concatenate([ctx,y[i*512:(i+1)*512]])[None].astype(np.float32); o,st=sess.run(None,{"input":xx,"state":st,"sr":np.array(16000,np.int64)}); sil.append(float(o[0,0])); ctx=xx[0,-64:]
+sil=np.array(sil); n=int(len(x)/160)
+S,_=fl.post(fl.hold(sil,0.032,n)>=0.5,**fl.POST_SIL)
+print('talk %.0f s; Silero speech time %.1f%%'%(len(x)/16000,100*S.mean()))
+for m in ['redux','ultra']:
+    z=logits_blocks(m,x); base,_=fl.post(fl.hold(sig(z),0.08,n)>=0.5,**fl.POST_HEAD)
+    def rc(q):
+        keep=np.zeros(len(z),bool)
+        for i,j in zip(*fl.runs(z>=0)):
+            if np.median(z[i:j])>=q: keep[i:j]=True
+        return fl.post(fl.hold(keep.astype(np.float32),0.08,n)>=0.5,**fl.POST_HEAD)[0]
+    print(f'\n{m}: speech time % | agreement with Silero (F1 of mask vs Silero mask) | frames removed vs head 0.5 (% of its speech frames)')
+    for nm,mk in [('head 0.5',base),('head 0.7',fl.post(fl.hold(sig(z),0.08,n)>=0.7,**fl.POST_HEAD)[0]),('head 0.9',fl.post(fl.hold(sig(z),0.08,n)>=0.9,**fl.POST_HEAD)[0]),('runconf 2.5',rc(2.5)),('runconf 3.5',rc(3.5))]:
+        P,R,F=fl.prf(fl.counts(mk,S)); print(f'  {nm:12s} {100*mk.mean():5.1f}% | F1 vs Silero {100*F:.1f} (P {100*P:.1f} R {100*R:.1f}) | removed {100*((base&~mk).sum()/base.sum()):.2f}%')
diff --git a/scripts/vad_bench/noise_dive/e_build.py b/scripts/vad_bench/noise_dive/e_build.py
new file mode 100644
index 0000000..e617e6e
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/e_build.py
@@ -0,0 +1,41 @@
+import os
+FUSION=os.environ.get('FUSION_DIR','../fusion')  # scripts/vad_bench/fusion with data/ built by prep_libri.py
+"""Rebuild the fusion synthetic set (same generators and seeds as fusion/mkcorpus.py + mkgap.py) and compute per-clip:
+head logits (redux, ultra F16; CPU reference validated against parakeet-cli), Silero probabilities (ONNX), and 10 ms frame energy."""
+import numpy as np,json,pickle,sys,os
+from vadlib import *; import corpus; import onnxruntime as ort
+FL=corpus.FL; libri=corpus.libri; spk=corpus.spk
+sets=corpus.speech_sets()
+conds=[("clean",None)]+[(k,s) for k in("white","pink") for s in(20,10,5,0)]
+Ps=[]; clips=[]
+q=lambda y: (np.round(np.clip(y,-1,1)*32767)/32767).astype(np.float32)   # PCM_16 round trip as in the original wavs
+for ci,(kind,snr) in enumerate(conds):
+    for k,(f,us) in enumerate(sets):
+        y,spans=FL.build(us,np.random.default_rng(1000+k))
+        if ci==0: Ps.append(float(np.mean(np.concatenate([y[int(s*16000):int(e*16000)] for s,e in spans])**2)))
+        y=FL.add_noise(y,spans,kind,snr,np.random.default_rng(5000+ci*100+k))
+        clips.append(dict(name=f"{kind}{'' if snr is None else snr}_{k:02d}",cond=f"{kind}{'' if snr is None else snr}",kind='speech',spans=spans,y=q(y)))
+for ci,(kind,snr) in enumerate(conds[1:]):
+    for k in range(12):
+        r=np.random.default_rng(9000+ci*100+k); P=Ps[(k*3)%len(Ps)]; n=30*16000
+        z=r.standard_normal(n) if kind=="white" else FL.pink(n,r)
+        z=np.clip(z*np.sqrt(P/10**(snr/10)),-1,1).astype(np.float32)
+        clips.append(dict(name=f"nz-{kind}{snr}_{k:02d}",cond=f"nz-{kind}{snr}",kind='noise',spans=[],y=q(z)))
+for g in corpus.gap_clips():
+    clips.append(dict(name=g['name'],cond=f"gap-{g['kind']}{g['snr']}",kind='gap',spans=g['spans'],stretch=g['stretch'],y=q(g['y'])))
+print(len(clips),'clips',sum(len(c['y']) for c in clips)/16000,'s',flush=True)
+so=ort.SessionOptions(); so.intra_op_num_threads=2; so.inter_op_num_threads=1
+sess=ort.InferenceSession(os.environ['SILERO_ONNX'],so,providers=["CPUExecutionProvider"])
+def silero(y):
+    n=(len(y)+511)//512; y=np.pad(y,(0,n*512-len(y))); st=np.zeros((2,1,128),np.float32); ctx=np.zeros(64,np.float32); out=[]
+    for i in range(n):
+        x=np.concatenate([ctx,y[i*512:(i+1)*512]])[None].astype(np.float32)
+        o,st=sess.run(None,{"input":x,"state":st,"sr":np.array(16000,np.int64)}); out.append(float(o[0,0])); ctx=x[0,-64:]
+    return np.array(out,np.float32)
+for i,c in enumerate(clips):
+    y=c['y']; c['z']={m:logits_blocks(m,y).astype(np.float32) for m in['redux','ultra']}
+    c['sil']=silero(y)
+    fr=len(y)//160; c['e10']=10*np.log10((y[:fr*160].reshape(fr,160).astype(np.float64)**2).mean(1)+1e-12)   # dBFS^2 per 10 ms
+    del c['y']
+    if i%40==0: print(i,flush=True)
+pickle.dump(clips,open('e_set.pkl','wb'))
diff --git a/scripts/vad_bench/noise_dive/e_eval.py b/scripts/vad_bench/noise_dive/e_eval.py
new file mode 100644
index 0000000..b6af397
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/e_eval.py
@@ -0,0 +1,60 @@
+import os
+FUSION=os.environ.get('FUSION_DIR','../fusion')  # scripts/vad_bench/fusion with data/ built by prep_libri.py
+import numpy as np,pickle,json,sys
+sys.path.insert(0,FUSION)
+import fl, rules
+from vadlib import sig
+GR=fl.GR
+clips=pickle.load(open('e_set.pkl','rb'))
+def grid(c):
+    n=int(round(len(c['e10'])))   # 10 ms cells
+    return n
+def refmask(c,n):
+    m=np.zeros(n,bool)
+    for s,e in c['spans']: m[int(round(s/GR)):int(round(e/GR))]=True
+    return m
+def stretch_mask(c,n):
+    m=np.zeros(n,bool)
+    if 'stretch' in c: a,b=c['stretch']; m[int(round(a/GR)):int(round(b/GR))]=True
+    return m
+def e80(c):
+    p=10**(c['e10']/10); k=np.ones(8)/8; return 10*np.log10(np.convolve(p,k,'same')+1e-12)
+def run_rule(rule,c,model):
+    n=len(c['e10']); H=fl.hold(sig(c['z'][model]),0.08,n); S=fl.hold(c['sil'],0.032,n)
+    kind=rule[0]
+    if kind=='head': m=H>=rule[1]; post=fl.POST_HEAD
+    elif kind=='head_ms': m=H>=0.5; post=dict(fl.POST_HEAD,min_speech=rule[1])
+    elif kind=='gate':   # head thr + relative energy gate
+        m=H>=rule[1]; e=e80(c); ref_lvl=np.percentile(c['e10'],95); m=m&(e>=ref_lvl-rule[2]); post=fl.POST_HEAD
+    elif kind=='runconf':   # keep a p>=0.5 run only when its median logit is >= Q (noise plateau sits near +1.5, speech runs near +7)
+        z=c['z'][model]; keep=np.zeros(len(z),bool); a=fl.runs(z>=0)
+        for i,j in zip(*a):
+            if np.median(z[i:j])>=rule[1]: keep[i:j]=True
+        m=fl.hold(keep.astype(np.float32),0.08,n)>=0.5; post=fl.POST_HEAD
+    elif kind=='silero': m=S>=0.5; post=fl.POST_SIL
+    elif kind=='and': m=(S>=0.5)&(H>=rule[1]); post=fl.POST_HEAD
+    elif kind=='two': m=rules.two_stage(S>=0.5,H>=0.5,16,16,30) if False else rules.two_stage(S>=0.5,H>=0.5,rule[1],rule[2],rule[3]); post=fl.POST_HEAD
+    return fl.post(m,**post)
+R=[('head',0.5),('head',0.7),('head',0.9),('head',0.97),('head_ms',0.25),('head_ms',0.5),('head_ms',1.0),
+   ('gate',0.5,15),('gate',0.5,25),('gate',0.5,35),('gate',0.9,25),
+   ('runconf',2.5),('runconf',3.5),('runconf',5.0),('and',0.5),('and',0.9),('two',16,16,30),('silero',)]
+def name(r):
+    k=r[0]; g=lambda i: r[i] if len(r)>i else ""
+    return {'head':lambda:f'head thr {g(1)}','head_ms':lambda:f'head thr 0.5, min_speech {g(1)} s','gate':lambda:f'head thr {g(1)} + energy gate (frame >= file P95 - {g(2)} dB)','and':lambda:f'Silero 0.5 AND head {g(1)}','two':lambda:'two-stage (Silero + head extend/fill, fusion rule 6)','silero':lambda:'Silero 0.5 alone (reference, its own post)','runconf':lambda:f'head thr 0.5, keep run only if median logit >= {g(1)}'}[k]()
+out={}
+for model in ['redux','ultra']:
+    print(f'\n### {model}\n| rule | speech F1 clean | speech F1 noisy (white+pink 20..0 dB) | noise-only FA frames % | embedded-stretch FA frames % | noise-only FA regions per clip |\n|---|---|---|---|---|---|')
+    for r in R:
+        cc={'clean':np.zeros(4),'noisy':np.zeros(4)}; fa_nz=[0,0]; fa_st=[0,0]; regs=[]
+        for c in clips:
+            n=len(c['e10']); m,rg=run_rule(r,c,model)
+            if c['kind']=='speech':
+                key='clean' if c['cond']=='clean' else 'noisy'; cc[key]+=fl.counts(m,refmask(c,n))
+            elif c['kind']=='noise': fa_nz[0]+=m.sum(); fa_nz[1]+=n; regs.append(len(rg))
+            else:
+                st=stretch_mask(c,n); fa_st[0]+=(m&st).sum(); fa_st[1]+=st.sum()
+        f=lambda k: 100*fl.prf(cc[k])[2]; P=lambda k:100*fl.prf(cc[k])[0]; Rc=lambda k:100*fl.prf(cc[k])[1]
+        out[f'{model}|{name(r)}']=dict(f1_clean=f('clean'),f1_noisy=f('noisy'),R_clean=Rc('clean'),P_clean=P('clean'),fa_noise=100*fa_nz[0]/fa_nz[1],fa_stretch=100*fa_st[0]/fa_st[1],regions=float(np.mean(regs)))
+        o=out[f'{model}|{name(r)}']
+        print(f'| {name(r)} | {o["f1_clean"]:.1f} (P {o["P_clean"]:.1f}, R {o["R_clean"]:.1f}) | {o["f1_noisy"]:.1f} | {o["fa_noise"]:.1f} | {o["fa_stretch"]:.1f} | {o["regions"]:.1f} |',flush=True)
+json.dump(out,open('res_E.json','w'),indent=1)
diff --git a/scripts/vad_bench/noise_dive/fig_D_S.png b/scripts/vad_bench/noise_dive/fig_D_S.png
new file mode 100644
index 0000000..8103002
Binary files /dev/null and b/scripts/vad_bench/noise_dive/fig_D_S.png differ
diff --git a/scripts/vad_bench/noise_dive/fig_energy.py b/scripts/vad_bench/noise_dive/fig_energy.py
new file mode 100644
index 0000000..b5038c2
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/fig_energy.py
@@ -0,0 +1,29 @@
+import os
+FUSION=os.environ.get('FUSION_DIR','../fusion')  # scripts/vad_bench/fusion with data/ built by prep_libri.py
+import numpy as np,pickle,sys
+sys.path.insert(0,FUSION)
+import matplotlib; matplotlib.use('Agg'); import matplotlib.pyplot as plt
+from vadlib import frame_mask
+clips=pickle.load(open('e_set.pkl','rb'))
+fig,axs=plt.subplots(1,2,figsize=(13,4.6),sharey=True)
+stats={}
+for ax,m in zip(axs,['redux','ultra']):
+    X={'speech frames (clean+noisy speech clips)':[],'non-speech frames in speech clips (gaps)':[],'noise-only files':[],'embedded noise stretch':[]}; Y={k:[] for k in X}
+    for c in clips:
+        z=c['z'][m]; n=len(z); e=c['e10'][:n*8].reshape(n,8) if len(c['e10'])>=n*8 else None
+        if e is None: n=len(c['e10'])//8; z=z[:n]; e=c['e10'][:n*8].reshape(n,8)
+        e80=10*np.log10((10**(e/10)).mean(1)+1e-12); rel=e80-np.percentile(c['e10'],95)
+        t=(np.arange(n)+.5)*.08
+        if c['kind']=='speech':
+            sp=frame_mask(c['spans'],n); X['speech frames (clean+noisy speech clips)'].append(rel[sp]); Y['speech frames (clean+noisy speech clips)'].append(z[sp])
+            X['non-speech frames in speech clips (gaps)'].append(rel[~sp]); Y['non-speech frames in speech clips (gaps)'].append(z[~sp])
+        elif c['kind']=='noise': X['noise-only files'].append(rel); Y['noise-only files'].append(z)
+        else:
+            a,b=c['stretch']; st=(t>=a)&(t speech frames %
+  redux 15.999999642372131 % ; mean logit -1.504223108291626 ; all-zero features logit 1.7988790273666382
+  ultra 58.399999141693115 % ; mean logit 0.17368166148662567 ; all-zero features logit 1.143395185470581
+B) offset d (in per-file std units) added to every bin and frame; scale s on the fluctuation (white noise normalised mel)
+redux
+   s\d  -3.0  -2.5  -2.0  -1.5  -1.0  -0.5   0.0   0.5   1.0   1.5   2.0   2.5   3.0
+  0.25     0     0     0     0     0     0     0     1    76    83    90    98   100
+  0.50     0     0     0     0     1    45    61    83    99    98    99    99   100
+  1.00     0     0     0     0     1    10    99    99    97    97    99    99    99
+  1.50     0     0     1    14    34    39    47    58    96    99    99    99    99
+  2.00    66    76    81    66    35     6     4    11    60    96    99    99    99
+ultra
+   s\d  -3.0  -2.5  -2.0  -1.5  -1.0  -0.5   0.0   0.5   1.0   1.5   2.0   2.5   3.0
+  0.25     0     0     0     0     0     0     0    77    61    14    38    49    60
+  0.50     0     0     0     0     0     0    14    94    82    43    20    22    42
+  1.00     0     0     0     0     6    81    99    97    66    18     5    11    23
+  1.50     0     1     6    30    77    95    94    66    14     8    20    33    51
+  2.00     1     2    10    30    49    47    20     3     1    11    38    58    64
+C) real speech normalised mel shifted by d: speech frames % (of frames)
+  redux [(-2, 55), (-1, 74), (-0.5, 90), (0, 91), (0.5, 92), (1, 91), (2, 98)]
+  ultra [(-2, 1), (-1, 44), (-0.5, 75), (0, 89), (0.5, 92), (1, 94), (2, 91)]
diff --git a/scripts/vad_bench/noise_dive/res_B5.txt b/scripts/vad_bench/noise_dive/res_B5.txt
new file mode 100644
index 0000000..dfee3cf
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/res_B5.txt
@@ -0,0 +1,34 @@
+| model | comparison | AUC | Youden-best logit thr (p) | TPR / FPR at best | F1-best logit thr (p) | speech/noise at default 0 |
+|---|---|---|---|---|---|---|
+| redux | speech (clean clips) vs pauses in the same clips | 0.990 | -3.36 (0.03) | 0.95 / 0.00 | -5.90 (0.00) | 92% / 0% |
+| redux | speech vs embedded noise stretch, SNR 20 dB | 0.948 | +1.72 (0.85) | 0.89 / 0.00 | +1.58 (0.83) | 93% / 33% |
+| redux | speech vs embedded noise stretch, SNR 10 dB | 0.931 | +1.82 (0.86) | 0.87 / 0.01 | +1.71 (0.85) | 94% / 60% |
+| redux | speech vs embedded noise stretch, SNR 5 dB | 0.947 | +1.71 (0.85) | 0.89 / 0.01 | +1.55 (0.82) | 95% / 56% |
+| redux | speech vs embedded noise stretch, SNR 0 dB | 0.943 | +1.67 (0.84) | 0.86 / 0.02 | +1.63 (0.84) | 95% / 55% |
+| redux | speech (clean) vs noise-only files -50..-15 dBFS pooled | 0.896 | +3.35 (0.97) | 0.85 / 0.02 | +3.14 (0.96) | 92% / 93% |
+| redux | speech (clean) vs noise-only file at -23 dBFS (speech-level) | 0.892 | +3.53 (0.97) | 0.85 / 0.01 | -13.09 (0.00) | 92% / 97% |
+| ultra | speech (clean clips) vs pauses in the same clips | 0.961 | -0.89 (0.29) | 0.92 / 0.00 | -1.86 (0.13) | 91% / 0% |
+| ultra | speech vs embedded noise stretch, SNR 20 dB | 0.982 | -1.85 (0.14) | 0.94 / 0.00 | -2.01 (0.12) | 91% / 0% |
+| ultra | speech vs embedded noise stretch, SNR 10 dB | 0.958 | +0.06 (0.51) | 0.89 / 0.01 | -0.08 (0.48) | 89% / 1% |
+| ultra | speech vs embedded noise stretch, SNR 5 dB | 0.956 | +1.10 (0.75) | 0.87 / 0.02 | +0.81 (0.69) | 92% / 13% |
+| ultra | speech vs embedded noise stretch, SNR 0 dB | 0.941 | +1.40 (0.80) | 0.85 / 0.03 | +1.29 (0.78) | 95% / 48% |
+| ultra | speech (clean) vs noise-only files -50..-15 dBFS pooled | 0.868 | +3.51 (0.97) | 0.81 / 0.01 | +3.01 (0.95) | 91% / 98% |
+| ultra | speech (clean) vs noise-only file at -23 dBFS (speech-level) | 0.865 | +3.59 (0.97) | 0.80 / 0.01 | -7.21 (0.00) | 91% / 99% |
+
+logit percentiles (5,25,50,75,95):
+redux  speech clean                                     [-3.13  5.52  7.97 10.01 12.67]
+redux  pause (digital -60 dBFS dither) in clean clips   [-11.63  -9.    -8.12  -7.26  -5.49]
+redux  noise-only -23 dBFS                              [0.16 0.89 1.45 2.03 2.99]
+redux  noise-only -40 dBFS                              [0.03 0.65 1.19 1.77 2.6 ]
+redux  noise-only -50 dBFS                              [-0.61  0.02  0.49  0.98  1.8 ]
+redux  embedded stretch 20 dB                           [-2.01 -1.   -0.35  0.19  0.91]
+redux  embedded stretch 5 dB                            [-0.84 -0.28  0.08  0.47  1.12]
+redux  embedded stretch 0 dB                            [-0.87 -0.33  0.08  0.53  1.26]
+ultra  speech clean                                     [-2.72  4.52  7.04  9.25 13.15]
+ultra  pause (digital -60 dBFS dither) in clean clips   [-4.99 -4.23 -3.23 -2.65 -1.76]
+ultra  noise-only -23 dBFS                              [0.5  1.23 1.74 2.27 3.08]
+ultra  noise-only -40 dBFS                              [0.37 1.09 1.57 2.08 2.86]
+ultra  noise-only -50 dBFS                              [0.05 0.76 1.25 1.78 2.55]
+ultra  embedded stretch 20 dB                           [-4.67 -4.06 -3.62 -3.17 -2.5 ]
+ultra  embedded stretch 5 dB                            [-2.41 -1.65 -1.08 -0.43  0.58]
+ultra  embedded stretch 0 dB                            [-1.3  -0.58 -0.04  0.48  1.24]
diff --git a/scripts/vad_bench/noise_dive/res_B6.txt b/scripts/vad_bench/noise_dive/res_B6.txt
new file mode 100644
index 0000000..1540e0f
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/res_B6.txt
@@ -0,0 +1,27 @@
+Embedded-stretch false-alarm frame % at threshold 0.5 ; rows=type, cols=SNR re speech (dB) ; 6 clips per cell
+
+redux:
+| type | -10 dB | 0 dB | 10 dB | 20 dB | 30 dB | 40 dB |
+|---|---|---|---|---|---|---|
+| white | 93 | 68 | 79 | 53 | 0 | 0 |
+| pink | 93 | 61 | 63 | 33 | 0 | 0 |
+| brown | 0 | 0 | 0 | 0 | 0 | 0 |
+| speech-shaped | 99 | 89 | 48 | 43 | 1 | 0 |
+| hum 50 Hz | 0 | 0 | 0 | 0 | 0 | 0 |
+| sine 1 kHz | 0 | 0 | 0 | 0 | 0 | 0 |
+| clicks 3/s | 2 | 1 | 1 | 1 | 0 | 0 |
+| clicks 10/s | 15 | 14 | 16 | 9 | 0 | 0 |
+| music-like | 30 | 23 | 16 | 10 | 1 | 0 |
+
+ultra:
+| type | -10 dB | 0 dB | 10 dB | 20 dB | 30 dB | 40 dB |
+|---|---|---|---|---|---|---|
+| white | 97 | 44 | 0 | 0 | 0 | 0 |
+| pink | 96 | 55 | 0 | 0 | 0 | 0 |
+| brown | 0 | 0 | 0 | 0 | 0 | 0 |
+| speech-shaped | 100 | 99 | 46 | 0 | 0 | 0 |
+| hum 50 Hz | 0 | 0 | 0 | 0 | 0 | 0 |
+| sine 1 kHz | 0 | 0 | 0 | 0 | 0 | 0 |
+| clicks 3/s | 46 | 38 | 26 | 9 | 0 | 0 |
+| clicks 10/s | 90 | 84 | 64 | 22 | 1 | 0 |
+| music-like | 55 | 53 | 44 | 8 | 0 | 0 |
diff --git a/scripts/vad_bench/noise_dive/res_B7.md b/scripts/vad_bench/noise_dive/res_B7.md
new file mode 100644
index 0000000..57044be
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/res_B7.md
@@ -0,0 +1,33 @@
+talk rms dBFS -20.4
+
+redux white: stretch FA % ; rows noise level re talk RMS, cols noise share of window (L s of 120)
+| level | 5 s (4%) | 10 s (8%) | 20 s (16%) | 30 s (25%) | 45 s (37%) | 60 s (50%) | 90 s (75%) |
+|---|---|---|---|---|---|---|---|
+| -35 dB | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
+| -25 dB | 0 | 0 | 0 | 0 | 0 | 0 | 8 |
+| -15 dB | 83 | 70 | 21 | 8 | 1 | 0 | 18 |
+| -5 dB | 22 | 12 | 6 | 3 | 1 | 1 | 17 |
+
+redux pink: stretch FA % ; rows noise level re talk RMS, cols noise share of window (L s of 120)
+| level | 5 s (4%) | 10 s (8%) | 20 s (16%) | 30 s (25%) | 45 s (37%) | 60 s (50%) | 90 s (75%) |
+|---|---|---|---|---|---|---|---|
+| -35 dB | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
+| -25 dB | 0 | 0 | 0 | 0 | 0 | 0 | 1 |
+| -15 dB | 0 | 0 | 0 | 0 | 0 | 0 | 12 |
+| -5 dB | 6 | 10 | 1 | 1 | 1 | 5 | 32 |
+
+ultra white: stretch FA % ; rows noise level re talk RMS, cols noise share of window (L s of 120)
+| level | 5 s (4%) | 10 s (8%) | 20 s (16%) | 30 s (25%) | 45 s (37%) | 60 s (50%) | 90 s (75%) |
+|---|---|---|---|---|---|---|---|
+| -35 dB | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
+| -25 dB | 85 | 98 | 87 | 26 | 1 | 0 | 0 |
+| -15 dB | 100 | 100 | 100 | 98 | 62 | 23 | 3 |
+| -5 dB | 100 | 100 | 94 | 76 | 60 | 62 | 64 |
+
+ultra pink: stretch FA % ; rows noise level re talk RMS, cols noise share of window (L s of 120)
+| level | 5 s (4%) | 10 s (8%) | 20 s (16%) | 30 s (25%) | 45 s (37%) | 60 s (50%) | 90 s (75%) |
+|---|---|---|---|---|---|---|---|
+| -35 dB | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
+| -25 dB | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
+| -15 dB | 57 | 42 | 24 | 8 | 2 | 0 | 1 |
+| -5 dB | 35 | 38 | 43 | 29 | 18 | 18 | 45 |
diff --git a/scripts/vad_bench/noise_dive/res_B8.txt b/scripts/vad_bench/noise_dive/res_B8.txt
new file mode 100644
index 0000000..b298dd9
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/res_B8.txt
@@ -0,0 +1,14 @@
+redux corr(D, median logit) 0.80 ; corr(D, FA) 0.45
+   D in [-2,-1.5): n= 10 FA mean   0.0%  median logit -5.68
+   D in [-1.5,-1): n= 28 FA mean   0.0%  median logit -6.59
+   D in [-1,-0.5): n= 55 FA mean   0.8%  median logit -4.64
+   D in [-0.5,0): n= 32 FA mean   1.5%  median logit -2.16
+   D in [0,0.5): n= 31 FA mean  14.1%  median logit -0.95
+   D in [0.5,9): n= 12 FA mean  21.9%  median logit -0.51
+ultra corr(D, median logit) 0.84 ; corr(D, FA) 0.64
+   D in [-2,-1.5): n= 10 FA mean   0.0%  median logit -5.27
+   D in [-1.5,-1): n= 28 FA mean   1.6%  median logit -4.67
+   D in [-1,-0.5): n= 55 FA mean   6.2%  median logit -3.91
+   D in [-0.5,0): n= 32 FA mean  19.2%  median logit -1.79
+   D in [0,0.5): n= 31 FA mean  46.9%  median logit -0.16
+   D in [0.5,9): n= 12 FA mean  83.1%  median logit +1.17
diff --git a/scripts/vad_bench/noise_dive/res_D.md b/scripts/vad_bench/noise_dive/res_D.md
new file mode 100644
index 0000000..408c4d9
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/res_D.md
@@ -0,0 +1,39 @@
+Inserted 60 s block at 111.7-171.7 s of a 300 s talk excerpt. Words whose start lies inside the block = hallucinated (the inserted audio has no speech).
+
+### ultra (talk excerpt alone: 811 words without VAD, 809 with VAD)
+| inserted audio | level re talk RMS | words in block, no VAD | words in block, --vad | talk words outside block, no VAD / --vad (base 811) | VAD segments touching block (s covered / count) |
+|---|---|---|---|---|---|
+| white | -35 dB | 0 | 0 | 811 / 812 | 54 s / 2 |
+| white | -20 dB | 0 | 0 | 811 / 809 | 60 s / 4 |
+| white | -5 dB | 0 | 0 | 811 / 809 | 60 s / 4 |
+| pink | -35 dB | 0 | 0 | 811 / 812 | 54 s / 2 |
+| pink | -20 dB | 0 | 0 | 811 / 808 | 54 s / 3 |
+| pink | -5 dB | 0 | 0 | 811 / 809 | 60 s / 4 |
+| clicks | -35 dB | 0 | 0 | 811 / 810 | 60 s / 4 |
+| clicks | -20 dB | 0 | 0 | 811 / 810 | 60 s / 4 |
+| clicks | -5 dB | 0 | 0 | 811 / 810 | 60 s / 4 |
+| music | -35 dB | 0 | 15 | 811 / 813 | 54 s / 2 |
+| music | -20 dB | 0 | 12 | 811 / 812 | 54 s / 2 |
+| music | -5 dB | 0 | 3 | 811 / 809 | 60 s / 4 |
+
+### redux (talk excerpt alone: 814 words without VAD, 812 with VAD)
+| inserted audio | level re talk RMS | words in block, no VAD | words in block, --vad | talk words outside block, no VAD / --vad (base 814) | VAD segments touching block (s covered / count) |
+|---|---|---|---|---|---|
+| white | -35 dB | 0 | 0 | 814 / 810 | 24 s / 2 |
+| white | -20 dB | 0 | 2 | 815 / 811 | 54 s / 2 |
+| white | -5 dB | 0 | 0 | 814 / 808 | 60 s / 4 |
+| pink | -35 dB | 0 | 0 | 814 / 810 | 24 s / 2 |
+| pink | -20 dB | 0 | 0 | 815 / 811 | 54 s / 2 |
+| pink | -5 dB | 0 | 0 | 813 / 814 | 60 s / 5 |
+| clicks | -35 dB | 0 | 0 | 813 / 809 | 60 s / 3 |
+| clicks | -20 dB | 0 | 0 | 814 / 810 | 60 s / 4 |
+| clicks | -5 dB | 0 | 0 | 815 / 813 | 60 s / 4 |
+| music | -35 dB | 0 | 0 | 814 / 813 | 60 s / 4 |
+| music | -20 dB | 0 | 0 | 814 / 810 | 60 s / 4 |
+| music | -5 dB | 0 | 0 | 815 / 813 | 60 s / 4 |
+
+example hallucinated text:
+ultra music_-35 | novad:  | vad: I don't know, I don't know, I don't know, I don't know,
+ultra music_-20 | novad:  | vad: I don't know if you're not sure if you're not a mistake.
+ultra music_-5 | novad:  | vad: I don't know.
+redux white_-20 | novad:  | vad: you're a
diff --git a/scripts/vad_bench/noise_dive/res_D_mit.md b/scripts/vad_bench/noise_dive/res_D_mit.md
new file mode 100644
index 0000000..5a1c81f
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/res_D_mit.md
@@ -0,0 +1,27 @@
+insert (60 s) frames called speech %, by mitigation (block-normalised logits, shipped segmenter smoothing not applied)
+| file | model | thr 0.5 | thr 0.7 | thr 0.9 | runconf 2.5 | talk speech frames kept (outside insert, vs thr 0.5) runconf 2.5 |
+|---|---|---|---|---|---|---|
+| white_-35 | redux | 0 | 0 | 0 | 0 | 99.9% |
+| white_-35 | ultra | 0 | 0 | 0 | 0 | 100.0% |
+| white_-20 | redux | 0 | 0 | 0 | 0 | 99.9% |
+| white_-20 | ultra | 27 | 13 | 4 | 0 | 99.9% |
+| white_-5 | redux | 3 | 0 | 0 | 0 | 99.9% |
+| white_-5 | ultra | 68 | 22 | 5 | 0 | 99.9% |
+| pink_-35 | redux | 0 | 0 | 0 | 0 | 99.9% |
+| pink_-35 | ultra | 0 | 0 | 0 | 0 | 100.0% |
+| pink_-20 | redux | 0 | 0 | 0 | 0 | 99.9% |
+| pink_-20 | ultra | 0 | 0 | 0 | 0 | 100.0% |
+| pink_-5 | redux | 5 | 0 | 0 | 0 | 99.9% |
+| pink_-5 | ultra | 23 | 1 | 0 | 0 | 100.0% |
+| clicks_-35 | redux | 1 | 1 | 0 | 0 | 99.8% |
+| clicks_-35 | ultra | 6 | 2 | 0 | 0 | 100.0% |
+| clicks_-20 | redux | 5 | 2 | 0 | 0 | 99.8% |
+| clicks_-20 | ultra | 25 | 18 | 8 | 3 | 100.0% |
+| clicks_-5 | redux | 3 | 1 | 0 | 0 | 99.9% |
+| clicks_-5 | ultra | 38 | 30 | 20 | 18 | 100.0% |
+| music_-35 | redux | 12 | 2 | 0 | 0 | 99.8% |
+| music_-35 | ultra | 0 | 0 | 0 | 0 | 100.0% |
+| music_-20 | redux | 3 | 0 | 0 | 0 | 99.8% |
+| music_-20 | ultra | 0 | 0 | 0 | 0 | 100.0% |
+| music_-5 | redux | 2 | 0 | 0 | 0 | 99.8% |
+| music_-5 | ultra | 38 | 27 | 10 | 7 | 100.0% |
diff --git a/scripts/vad_bench/noise_dive/res_D_probs.md b/scripts/vad_bench/noise_dive/res_D_probs.md
new file mode 100644
index 0000000..db33199
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/res_D_probs.md
@@ -0,0 +1,28 @@
+| file | model | insert frames called speech %: block-120 s norm (shipped) | whole-file norm | fixed norm A | median logit (shipped) |
+|---|---|---|---|---|---|
+| base | redux | 99 | 100 | 100 | +6.6 |
+| base | ultra | 99 | 99 | 99 | +7.0 |
+| white_-35 | redux | 0 | 0 | 0 | -8.0 |
+| white_-35 | ultra | 0 | 0 | 0 | -3.5 |
+| white_-20 | redux | 0 | 2 | 0 | -2.1 |
+| white_-20 | ultra | 27 | 100 | 13 | -0.4 |
+| white_-5 | redux | 3 | 3 | 96 | -1.0 |
+| white_-5 | ultra | 68 | 87 | 100 | +0.3 |
+| pink_-35 | redux | 0 | 0 | 0 | -9.4 |
+| pink_-35 | ultra | 0 | 0 | 0 | -5.0 |
+| pink_-20 | redux | 0 | 0 | 0 | -3.3 |
+| pink_-20 | ultra | 0 | 0 | 0 | -1.9 |
+| pink_-5 | redux | 5 | 2 | 24 | -1.0 |
+| pink_-5 | ultra | 23 | 26 | 73 | -0.4 |
+| clicks_-35 | redux | 1 | 0 | 0 | -6.7 |
+| clicks_-35 | ultra | 6 | 2 | 0 | -3.6 |
+| clicks_-20 | redux | 5 | 2 | 1 | -5.5 |
+| clicks_-20 | ultra | 25 | 19 | 14 | -2.2 |
+| clicks_-5 | redux | 3 | 2 | 1 | -5.8 |
+| clicks_-5 | ultra | 38 | 32 | 27 | -0.9 |
+| music_-35 | redux | 12 | 0 | 0 | -2.8 |
+| music_-35 | ultra | 0 | 0 | 0 | -5.3 |
+| music_-20 | redux | 3 | 0 | 0 | -3.2 |
+| music_-20 | ultra | 0 | 0 | 0 | -3.8 |
+| music_-5 | redux | 2 | 0 | 0 | -3.6 |
+| music_-5 | ultra | 38 | 32 | 43 | -1.3 |
diff --git a/scripts/vad_bench/noise_dive/res_D_silero.txt b/scripts/vad_bench/noise_dive/res_D_silero.txt
new file mode 100644
index 0000000..3b532db
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/res_D_silero.txt
@@ -0,0 +1,5 @@
+Silero VAD on the 300 s talk excerpt with a 60 s inserted block at 111.7 to 171.7 s.
+Segments from the Silero run (files white_-20 and music_-35; the other files were not run with Silero):
+  segment 99.968 to 129.952 s is the only one touching the block, so 18.25 s of the block are kept
+  and 41.75 s are dropped.
+Transcription with the Ultra model and the Silero segments, music at -35 dB: 20 words start inside the block.
diff --git a/scripts/vad_bench/noise_dive/res_E.md b/scripts/vad_bench/noise_dive/res_E.md
new file mode 100644
index 0000000..f855f0f
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/res_E.md
@@ -0,0 +1,44 @@
+
+### redux
+| rule | speech F1 clean | speech F1 noisy (white+pink 20..0 dB) | noise-only FA frames % | embedded-stretch FA frames % | noise-only FA regions per clip |
+|---|---|---|---|---|---|
+| head thr 0.5 | 95.4 (P 98.6, R 92.3) | 94.7 | 97.8 | 55.2 | 2.1 |
+| head thr 0.7 | 94.8 (P 98.9, R 91.0) | 93.9 | 76.5 | 10.1 | 13.8 |
+| head thr 0.9 | 93.9 (P 99.3, R 89.1) | 92.0 | 16.3 | 0.1 | 12.9 |
+| head thr 0.97 | 92.5 (P 99.5, R 86.4) | 89.4 | 1.0 | 0.0 | 1.1 |
+| head thr 0.5, min_speech 0.25 s | 95.4 (P 98.7, R 92.3) | 94.7 | 97.7 | 51.7 | 2.2 |
+| head thr 0.5, min_speech 0.5 s | 95.4 (P 98.7, R 92.3) | 94.9 | 97.4 | 43.3 | 2.2 |
+| head thr 0.5, min_speech 1.0 s | 94.6 (P 98.7, R 90.8) | 94.1 | 96.6 | 29.0 | 2.2 |
+| head thr 0.5 + energy gate (frame >= file P95 - 15 dB) | 86.1 (P 100.0, R 75.6) | 91.7 | 97.8 | 40.5 | 2.1 |
+| head thr 0.5 + energy gate (frame >= file P95 - 25 dB) | 92.1 (P 99.9, R 85.4) | 94.3 | 97.8 | 53.2 | 2.1 |
+| head thr 0.5 + energy gate (frame >= file P95 - 35 dB) | 94.6 (P 99.7, R 90.1) | 94.7 | 97.8 | 55.2 | 2.1 |
+| head thr 0.9 + energy gate (frame >= file P95 - 25 dB) | 91.4 (P 99.9, R 84.1) | 91.8 | 16.3 | 0.1 | 12.9 |
+| head thr 0.5, keep run only if median logit >= 2.5 | 95.4 (P 98.7, R 92.3) | 94.7 | 0.1 | 0.3 | 0.0 |
+| head thr 0.5, keep run only if median logit >= 3.5 | 94.9 (P 98.7, R 91.4) | 93.7 | 0.0 | 0.2 | 0.0 |
+| head thr 0.5, keep run only if median logit >= 5.0 | 93.5 (P 98.7, R 88.8) | 90.2 | 0.0 | 0.1 | 0.0 |
+| Silero 0.5 AND head 0.5 | 93.4 (P 99.4, R 88.2) | 91.8 | 0.0 | 0.0 | 0.0 |
+| Silero 0.5 AND head 0.9 | 93.0 (P 99.5, R 87.4) | 90.8 | 0.0 | 0.0 | 0.0 |
+| two-stage (Silero + head extend/fill, fusion rule 6) | 95.4 (P 98.6, R 92.4) | 94.8 | 0.0 | 0.0 | 0.0 |
+| Silero 0.5 alone (reference, its own post) | 93.9 (P 99.1, R 89.2) | 92.4 | 0.0 | 0.0 | 0.0 |
+
+### ultra
+| rule | speech F1 clean | speech F1 noisy (white+pink 20..0 dB) | noise-only FA frames % | embedded-stretch FA frames % | noise-only FA regions per clip |
+|---|---|---|---|---|---|
+| head thr 0.5 | 95.0 (P 98.5, R 91.7) | 93.6 | 99.4 | 17.3 | 1.1 |
+| head thr 0.7 | 94.2 (P 98.8, R 90.0) | 91.8 | 94.2 | 3.6 | 5.1 |
+| head thr 0.9 | 92.4 (P 99.2, R 86.5) | 87.5 | 25.8 | 0.0 | 18.2 |
+| head thr 0.97 | 90.0 (P 99.5, R 82.1) | 80.6 | 0.5 | 0.0 | 0.7 |
+| head thr 0.5, min_speech 0.25 s | 95.0 (P 98.5, R 91.7) | 93.6 | 99.4 | 15.4 | 1.1 |
+| head thr 0.5, min_speech 0.5 s | 95.0 (P 98.5, R 91.7) | 93.5 | 99.4 | 11.9 | 1.1 |
+| head thr 0.5, min_speech 1.0 s | 93.8 (P 98.6, R 89.6) | 91.9 | 99.4 | 6.9 | 1.1 |
+| head thr 0.5 + energy gate (frame >= file P95 - 15 dB) | 86.0 (P 100.0, R 75.4) | 90.9 | 99.4 | 17.3 | 1.1 |
+| head thr 0.5 + energy gate (frame >= file P95 - 25 dB) | 92.0 (P 99.9, R 85.2) | 93.3 | 99.4 | 17.3 | 1.1 |
+| head thr 0.5 + energy gate (frame >= file P95 - 35 dB) | 94.4 (P 99.7, R 89.6) | 93.6 | 99.4 | 17.3 | 1.1 |
+| head thr 0.9 + energy gate (frame >= file P95 - 25 dB) | 90.4 (P 99.9, R 82.5) | 87.3 | 25.8 | 0.0 | 18.2 |
+| head thr 0.5, keep run only if median logit >= 2.5 | 94.5 (P 98.5, R 90.8) | 90.3 | 0.0 | 0.0 | 0.0 |
+| head thr 0.5, keep run only if median logit >= 3.5 | 93.6 (P 98.6, R 89.0) | 85.3 | 0.0 | 0.0 | 0.0 |
+| head thr 0.5, keep run only if median logit >= 5.0 | 89.5 (P 98.6, R 82.0) | 67.2 | 0.0 | 0.0 | 0.0 |
+| Silero 0.5 AND head 0.5 | 93.3 (P 99.4, R 87.8) | 91.2 | 0.0 | 0.0 | 0.0 |
+| Silero 0.5 AND head 0.9 | 91.7 (P 99.5, R 85.0) | 86.7 | 0.0 | 0.0 | 0.0 |
+| two-stage (Silero + head extend/fill, fusion rule 6) | 95.1 (P 98.5, R 92.0) | 94.2 | 0.0 | 0.0 | 0.0 |
+| Silero 0.5 alone (reference, its own post) | 93.9 (P 99.1, R 89.2) | 92.4 | 0.0 | 0.0 | 0.0 |
diff --git a/scripts/vad_bench/noise_dive/res_E2_talk.txt b/scripts/vad_bench/noise_dive/res_E2_talk.txt
new file mode 100644
index 0000000..0269f54
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/res_E2_talk.txt
@@ -0,0 +1,15 @@
+talk 1168 s; Silero speech time 93.4%
+
+redux: speech time % | agreement with Silero (F1 of mask vs Silero mask) | frames removed vs head 0.5 (% of its speech frames)
+  head 0.5      97.0% | F1 vs Silero 98.0 (P 96.2 R 99.9) | removed 0.00%
+  head 0.7      95.9% | F1 vs Silero 98.4 (P 97.1 R 99.7) | removed 1.05%
+  head 0.9      94.1% | F1 vs Silero 98.5 (P 98.1 R 98.8) | removed 2.93%
+  runconf 2.5   96.7% | F1 vs Silero 98.1 (P 96.5 R 99.9) | removed 0.29%
+  runconf 3.5   96.5% | F1 vs Silero 98.1 (P 96.5 R 99.7) | removed 0.50%
+
+ultra: speech time % | agreement with Silero (F1 of mask vs Silero mask) | frames removed vs head 0.5 (% of its speech frames)
+  head 0.5      96.3% | F1 vs Silero 98.3 (P 96.8 R 99.8) | removed 0.00%
+  head 0.7      95.1% | F1 vs Silero 98.6 (P 97.7 R 99.5) | removed 1.24%
+  head 0.9      92.9% | F1 vs Silero 98.4 (P 98.6 R 98.1) | removed 3.57%
+  runconf 2.5   96.1% | F1 vs Silero 98.3 (P 97.0 R 99.8) | removed 0.22%
+  runconf 3.5   96.0% | F1 vs Silero 98.3 (P 97.0 R 99.7) | removed 0.28%
diff --git a/scripts/vad_bench/noise_dive/res_cli_check.txt b/scripts/vad_bench/noise_dive/res_cli_check.txt
new file mode 100644
index 0000000..2d9c357
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/res_cli_check.txt
@@ -0,0 +1,17 @@
+gap-white20_00 redux stretch FP pct CLI 42.5 ref 42.5 | max|dp| 0.0003
+gap-white20_00 ultra stretch FP pct CLI 0.0 ref 0.0 | max|dp| 0.0075
+gap-white20_01 redux stretch FP pct CLI 57.4 ref 57.4 | max|dp| 0.0003
+gap-white20_01 ultra stretch FP pct CLI 0.0 ref 0.0 | max|dp| 0.0071
+gap-white5_00 redux stretch FP pct CLI 71.5 ref 71.5 | max|dp| 0.0002
+gap-white5_00 ultra stretch FP pct CLI 2.3 ref 2.0 | max|dp| 0.0119
+gap-white5_01 redux stretch FP pct CLI 56.4 ref 56.4 | max|dp| 0.0369
+gap-white5_01 ultra stretch FP pct CLI 0.8 ref 0.8 | max|dp| 0.0324
+gap-white0_00 redux stretch FP pct CLI 50.0 ref 50.0 | max|dp| 0.0067
+gap-white0_00 ultra stretch FP pct CLI 32.0 ref 31.5 | max|dp| 0.0167
+gap-white0_01 redux stretch FP pct CLI 40.8 ref 40.8 | max|dp| 0.0002
+gap-white0_01 ultra stretch FP pct CLI 19.4 ref 18.9 | max|dp| 0.0086
+amp -20 {'redux': np.float64(97.6), 'ultra': np.float64(98.9)}
+amp -50 {'redux': np.float64(69.6), 'ultra': np.float64(97.9)}
+amp -60 {'redux': np.float64(22.4), 'ultra': np.float64(85.1)}
+amp -70 {'redux': np.float64(8.0), 'ultra': np.float64(42.9)}
+amp -80 {'redux': np.float64(0.0), 'ultra': np.float64(6.9)}
diff --git a/scripts/vad_bench/noise_dive/res_energy_corr.txt b/scripts/vad_bench/noise_dive/res_energy_corr.txt
new file mode 100644
index 0000000..a0652ac
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/res_energy_corr.txt
@@ -0,0 +1,8 @@
+('redux', 'speech frames (clean+noisy speech clips)') corr(rel energy, logit) 0.46 over 152550 frames
+('redux', 'non-speech frames in speech clips (gaps)') corr(rel energy, logit) 0.71 over 28764 frames
+('redux', 'noise-only files') corr(rel energy, logit) -0.02 over 36000 frames
+('redux', 'embedded noise stretch') corr(rel energy, logit) 0.31 over 37043 frames
+('ultra', 'speech frames (clean+noisy speech clips)') corr(rel energy, logit) 0.40 over 152550 frames
+('ultra', 'non-speech frames in speech clips (gaps)') corr(rel energy, logit) 0.43 over 28764 frames
+('ultra', 'noise-only files') corr(rel energy, logit) 0.05 over 36000 frames
+('ultra', 'embedded noise stretch') corr(rel energy, logit) 0.84 over 37043 frames
diff --git a/scripts/vad_bench/noise_dive/sig.py b/scripts/vad_bench/noise_dive/sig.py
new file mode 100644
index 0000000..cff91e0
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/sig.py
@@ -0,0 +1,38 @@
+import numpy as np
+SR=16000
+def rms(x): return float(np.sqrt(np.mean(np.square(x,dtype=np.float64))))
+def at_db(x,db):  # set RMS to db dBFS
+    return (x/ (rms(x)+1e-12) * 10**(db/20)).astype(np.float32)
+def white(n,r): return r.standard_normal(n)
+def colored(n,r,alpha):  # 1/f^alpha power spectrum  (alpha=1 pink, 2 brown)
+    X=np.fft.rfft(r.standard_normal(n)); f=np.arange(len(X),dtype=float); f[0]=1
+    y=np.fft.irfft(X/f**(alpha/2),n); return y
+def shaped(n,r,spec):  # noise with a given long-term magnitude spectrum (len n//2+1 interpolated)
+    X=np.fft.rfft(r.standard_normal(n)); f=np.linspace(0,1,len(X)); sp=np.interp(f,np.linspace(0,1,len(spec)),spec)
+    return np.fft.irfft(X*sp,n)
+def ltass(utts,nfft=1024):
+    acc=np.zeros(nfft//2+1)
+    for u in utts:
+        for i in range(0,len(u)-nfft,nfft//2):
+            acc+=np.abs(np.fft.rfft(u[i:i+nfft]*np.hanning(nfft)))**2
+    return np.sqrt(acc/acc.sum())
+def sine(n,f,db=-20): t=np.arange(n)/SR; return at_db(np.sin(2*np.pi*f*t),db)
+def sweep(n,f0=50,f1=7500,db=-20):
+    t=np.arange(n)/SR; T=n/SR; ph=2*np.pi*(f0*t+(f1-f0)*t**2/(2*T)); return at_db(np.sin(ph),db)
+def clicks(n,r,rate=3.0,db=-12,decay_ms=8):  # claps/coughs-like: decaying noise bursts at Poisson times
+    y=np.zeros(n); L=int(SR*decay_ms/1000*6)
+    env=np.exp(-np.arange(L)/(SR*decay_ms/1000))
+    for t in np.cumsum(r.exponential(1/rate,int(n/SR*rate*2))):
+        i=int(t*SR)
+        if i+L>=n: break
+        y[i:i+L]+=r.standard_normal(L)*env
+    return at_db(y,db)*1.0
+def hum(n,f=50,db=-25):
+    t=np.arange(n)/SR; y=sum(np.sin(2*np.pi*f*k*t)/k for k in (1,2,3,4,5)); return at_db(y,db)
+def music(n,r,db=-20):  # chords of harmonic tones, changing every 0.5 s
+    y=np.zeros(n); seg=SR//2; roots=[220,247,262,294,330,349,392]
+    for i in range(0,n,seg):
+        f0=r.choice(roots)*r.choice([0.5,1,2]); L=min(seg,n-i); t=np.arange(L)/SR
+        v=sum(np.sin(2*np.pi*f0*m*t)/m for m in (1,2,3,4,5)) + sum(np.sin(2*np.pi*f0*1.26*m*t)/m for m in (1,2,3))
+        env=np.minimum(1,np.minimum(t/0.02,(L/SR-t)/0.05)); y[i:i+L]=v*env
+    return at_db(y,db)
diff --git a/scripts/vad_bench/noise_dive/vadlib.py b/scripts/vad_bench/noise_dive/vadlib.py
new file mode 100644
index 0000000..54a281b
--- /dev/null
+++ b/scripts/vad_bench/noise_dive/vadlib.py
@@ -0,0 +1,94 @@
+"""Independent reference of the Moondream Parakeet VAD head path, built on HF transformers
+(ParakeetFeatureExtractor for log-mel, ParakeetEncoderSubsamplingConv2D for the 8x subsampler)
+plus the head structure documented in parakeet.cpp docs/ternary.md. Weights come from the HF safetensors."""
+import json, os, subprocess, numpy as np, torch, soundfile as sf
+torch.set_num_threads(4)
+from transformers.models.parakeet.configuration_parakeet import ParakeetEncoderConfig
+from transformers.models.parakeet.modeling_parakeet import ParakeetEncoderSubsamplingConv2D
+from transformers.models.parakeet.feature_extraction_parakeet import ParakeetFeatureExtractor
+HERE = os.environ.get("NOISE_DIVE_WORK", os.path.dirname(os.path.abspath(__file__)))  # holds build/, ref/, hfmeta/
+CLI = os.environ.get("PARAKEET_CLI", f"{HERE}/build/examples/cli/parakeet-cli")
+SCR = os.environ.get("GGUF_ROOT", os.path.dirname(HERE))  # holds gguf/ and vad-card/
+GG = {'redux': f"{SCR}/vad-card/redux-vad.gguf", 'ultra': f"{SCR}/vad-card/ultra-vad-q8_0.gguf", 'ultra-f16': f"{SCR}/gguf/ultra-f16.gguf", 'redux-keep': f"{SCR}/gguf/redux-keep.gguf"}
+FE = ParakeetFeatureExtractor(feature_size=128)
+EPS = 1e-5
+
+def raw_logmel(x):
+    """[T,128] un-normalised log-mel of a float32 mono 16 kHz array (HF code path)."""
+    w = torch.tensor(np.asarray(x, np.float32))[None, :]
+    pre = torch.cat([w[:, :1], w[:, 1:] - FE.preemphasis * w[:, :-1]], 1)
+    return FE._torch_extract_fbank_features(pre)[0]
+
+def norm_mel(m, mean=None, std=None, f64=True):
+    """per-feature (over time) mean/std normalisation as HF/NeMo (HF masks the last STFT frame: stats over n_samples//160
+    frames); or fixed mean/std (len-128 tensors). f64=True accumulates in double like parakeet.cpp; HF uses float32."""
+    n = m.shape[0] - 1
+    d = m.double() if f64 else m
+    if mean is None:
+        mv = d[:n]
+        mean = mv.sum(0) / n
+        std = torch.sqrt(((mv - mean) ** 2).sum(0) / (n - 1))
+    else:
+        mean, std = mean.to(d.dtype), std.to(d.dtype)
+    out = ((d - mean) / (std + EPS)).float()
+    return out[:n]     # the masked last STFT frame is dropped (parakeet.cpp keeps n_samples//160 frames)
+
+class Ref:
+    def __init__(self, name):
+        self.name = name
+        W = np.load(f"{HERE}/ref/{name}_weights.npz")
+        cfg = ParakeetEncoderConfig(**{k: v for k, v in json.load(open(f"{HERE}/hfmeta/{name}.config.json"))['encoder_config'].items() if k != 'model_type'})
+        self.sub = ParakeetEncoderSubsamplingConv2D(cfg).eval()
+        sd = {k[len('encoder.subsampling.'):]: torch.tensor(v.astype(np.float32)) for k, v in W.items() if k.startswith('encoder.subsampling.')}
+        self.sub.load_state_dict(sd)
+        g = lambda k: torch.tensor(W[k].astype(np.float32))
+        self.pw, self.pb = g('vad_head.proj.weight')[:, :, 0], g('vad_head.proj.bias')
+        self.cw, self.cb = g('vad_head.ctx.weight'), g('vad_head.ctx.bias')
+        self.ow, self.ob = g('vad_head.out.weight')[0, :, 0], g('vad_head.out.bias')[0]
+    @torch.no_grad()
+    def sub_out(self, mel_norm):
+        return self.sub(mel_norm[None])[0]            # [T/8,1024]
+    @torch.no_grad()
+    def head_logits(self, s):
+        h = torch.nn.functional.silu(s @ self.pw.T + self.pb)          # [T,128]
+        c = torch.nn.functional.conv1d(h.T[None], self.cw, self.cb, padding=2)[0].T
+        c = torch.nn.functional.silu(c)
+        return c @ self.ow + self.ob                                   # logits [T]
+    def logits(self, x, mean=None, std=None):
+        return self.head_logits(self.sub_out(norm_mel(raw_logmel(x), mean, std))).numpy()
+    def probs(self, x, **kw):
+        return 1 / (1 + np.exp(-self.logits(x, **kw)))
+    def feats(self, x, mean=None, std=None):
+        return self.sub_out(norm_mel(raw_logmel(x), mean, std)).numpy()
+
+_refs = {}
+def ref(name):
+    if name not in _refs: _refs[name] = Ref(name)
+    return _refs[name]
+
+def cli_probs(name, wav, thr=None):
+    out = subprocess.run([CLI, 'vad', '--model', GG[name], '--input', wav, '--probabilities', '--threads', '4'], capture_output=True, text=True)
+    j = json.loads(out.stdout)
+    return np.array(j['probabilities'], np.float32)
+
+def wr(path, x):
+    sf.write(path, np.clip(x, -1, 1).astype(np.float32), 16000, subtype='FLOAT')
+
+BLOCK = 120 * 16000
+def logits_blocks(name, x, mean=None, std=None, block=BLOCK, tail=5 * 16000):
+    """logits with parakeet.cpp's 120 s blocks (own mel normalisation per block), or fixed mean/std if given."""
+    r = ref(name); out = []; pos = 0; fs = 1280
+    while True:
+        ln = min(block, len(x) - pos)
+        if len(x) - pos - ln < tail: ln = len(x) - pos
+        z = r.logits(x[pos:pos + ln], mean, std)
+        last = pos + ln >= len(x)
+        if not last: z = z[:block // fs]
+        out.append(z); pos += ln
+        if pos >= len(x): break
+    return np.concatenate(out)
+def sig(z): return 1 / (1 + np.exp(-z))
+def frame_mask(spans, n, fs=0.08):
+    t = (np.arange(n) + 0.5) * fs; m = np.zeros(n, bool)
+    for a, b in spans: m |= (t >= a) & (t < b)
+    return m