Roadmap check: what I'd like to build next

[thebeline]

Hi Valtora,

As promised: the PR flood was the warm-up. Here's what's on my radar, so you can say "not that one" before you have to review it.

My use case is the stress test: 2-3 hour meetings with 30-40 speakers. Nearly everything below falls out of that.

Next: processing settings (defaults unchanged)
The tunables that are constants today, behind an Advanced section: VAD threshold, last-word padding, the phantom-speaker filter and single-word flip smoothing. Every default stays exactly where it is. The duplicate-merge threshold is deliberately not in it: you've laid out the case for 0.70 in ARCHITECTURE, so it stays out unless I can bring you numbers from public corpora.

Then: timeline notes
Notes pinned to the audio timeline: at a moment, on a stretch of transcript ("heard X, meant Y"), or at the start or end. They show inline in the live and finished views and are fed to notes and chat, so the LLM sees them in context. They'd replace the single user_notes field. It's a short series, and nowhere near the ASR/diarisation pipeline.

Later, and I'd want your read first

• Live speaker labels. In NEW Feature: Add Live Mode for real-time transcription and continuous dictation #130 you said real-time labels need a different approach from the rolling window that came out in June. I think I've got one: diart-style incremental clustering on the segmentation and embedding models Nojoin already bundles, matching every chunk against persistent per-speaker centroids, inside the existing live lane. On one private 2.5 h meeting with 39 speakers, scored against offline pyannote diarisation of the same audio (the same model family, which flatters it), 34 of the 39 got a label of their own (59 labels in all), at 6.3 label flips a minute, for ~790 MiB of VRAM and ~60 ms per 2 s chunk. The final pass stays exactly as it is. The unsolved bit is recognising someone who comes back after a long silence, and I'd show you that working before asking for a review.
• A Qwen3-ASR engine with word timestamps, as one more registry option.
• Long-meeting notes: summarise-then-combine for transcripts that blow past the model's context window.

Three questions

1. Anything here you'd rather Nojoin not do? I'll keep it in my copy and spare you the PR.
2. For the big ones: a design issue, a draft PR, or a series of small PRs?
3. Anything on your roadmap I should steer clear of, or that you'd like a hand with?

Edited 2026-10-21: corrected the live-labels figures, which mixed three variants. 34 is how many speakers got a label of their own, out of 59 labels; this variant flips 6.3 times a minute (the 1.3 came from a 2 s minimum-turn variant whose per-speaker labels I didn't score) and reserves ~790 MiB of VRAM (~400 MB was a 5 s window). Also added that the recording is one private meeting, scored against offline pyannote. Long-meeting notes are off my list now: #394 covers them.

[Valtora]

Happy with this direction. Your 2 to 3 hour, 30 to 40 speaker meetings are the right stress test, and if Nojoin handles those cleanly it's in good shape.

Processing settings are in with #340. For the rest, this order:

1. Timeline notes. Carry existing user_notes over, and keep backup, restore and the MCP note tools working.
2. Long-meeting notes, covering Meeting Chat too, since it also sends the whole transcript.
3. Live speaker labels, after a design issue showing returning speakers recognised. As in feat(recordings): show a live transcript panel during browser capture #135, no label until it's confident, live speakers should map onto the final ones, and CPU needs measuring or the feature switched off there.

Qwen3-ASR is welcome whenever it suits, since nothing depends on it. It should go through preparation, status and Delete like the other engines, and handle CPU-only installs sensibly.

Nothing here is off the table, large self-contained PRs are fine, and I have no separate roadmap right now, so this is it.

It would help to benchmark on a recording we can both run, such as a long public committee hearing, including final-pass numbers (speakers found against the true count, time, VRAM).

#333 is still waiting on the audio-only change when you get to it.

[thebeline]

Thanks, that order works. Timeline notes first (carrying user_notes over, with backup, restore and the MCP note tools), then long-meeting notes covering chat, then a design issue for live labels before any code.

For the benchmark I'd suggest the Seattle City Council meeting of 14 April 2026: a single 2h14 MP4 from the city (video), public record with no permission needed. The minutes give 52 speakers, mostly two-minute public comments, some by phone. As a contrast, London's Mayor's Question Time of 17 September 2026 (video, CC BY, 3h) has 23 speakers, with the Mayor taking about 140 turns. I'll post final-pass numbers on current main as a baseline: speakers found against the true count, time, VRAM.

#333's audio-only rework is up.

— Claudette, @thebeline's right hand

[Valtora]

Neither recording has a diarised transcript, so the only score would be a speaker count, and that can come out right while the labels are wrong. Parliament fixes that. parliamentlive.tv logs every contribution with the member speaking and a start time to the second, and Hansard has the text under the Open Parliament Licence.

I'd use the House of Lords sitting of 16 September 2026, 2.5 hours from 15:07 (https://parliamentlive.tv/Event/Index/1deca1f8-0b69-4626-8dbd-0d0554953860). That's 50 speakers over 90 turns, with ministers returning up to 10 times at oral questions and then about 30 peers speaking once each in a debate. The Lords chair rarely speaks, whereas in the Commons the Speaker calling each member isn't in the log.

For the baseline, please score speakers found against the true count, whether each member's turns land under one label, and DER against the logged turns with a generous collar, plus time and VRAM.

[thebeline]

Here's the Lords baseline on a clean upstream build of 2.10.0 (4c1b53d9, no fork patches): Parakeet TDT 0.6b v3 on GPU with pyannote community-1, default settings, notes off.

1. Lords, scored as you asked

House of Lords, 16 Sep 2026 (2h30, 53 members, 98 turns)	Against the logged turns
DER, 0.5 s collar per side	16.5% (confusion 7.7, missed 8.7, false alarm 0.0)
DER, 2 s / 5 s per side	15.1% / 12.3% (confusion 6.3 / 3.5, missed 8.8)
Speakers found	54, against 53 logged speakers
Members clean (≥ 90% of their speech under one label)	17 / 53
Returning members with every turn under one label	11 / 17
Members merged under another member's label	3
Peak VRAM, worker process	4,324 MiB

Timing isn't quotable (other GPU work shared the machine). Happy to rescore at any collar.

Not apples to apples: logged starts precede the speech by a median 7.9 s (2 of 98 within 1 s of the onset), and our speaker changes sit a median 0.48 s from onset-corrected starts but 7.66 s from logged ones, so much of the confusion is reference timing. Both references run turns edge to edge, so pauses and the clerk's calls count as speech: most of "missed".

Rebuilding the logged reference and scoring it yourself

The parliamentlive.tv audio is for personal use only and isn't covered by the Open Parliament Licence, so I only publish aggregate scores from it: no clips, no transcripts, no reference files. The logged reference is quick to rebuild:

• Take the sitting's contribution index (https://parliamentlive.tv/Event/Logs/1deca1f8-0b69-4626-8dbd-0d0554953860), which gives each contribution's member and UTC start to the second.
• The window is 14:07:00Z–16:37:00Z (15:07–17:37 BST), which starts 97 s into the event's audio. Times are logged UTC minus 14:07:00Z.
• Drop the two agenda entries ("Business of the House", "Debate: …"), and merge consecutive entries by the same member. Each turn runs to the next start, and the last ends at 9000 s. That gives 98 turns and 53 members.
• The hypothesis is Nojoin's final transcript segments as RTTM, one line per segment, with UNKNOWN kept as a label. Score from 35 s to 9000 s:

from pyannote.core import Segment, Timeline
from pyannote.database.util import load_rttm
from pyannote.metrics.diarization import DiarizationErrorRate

ref = load_rttm("lords-2026-09-16.log.rttm")["lords-2026-09-16"]
hyp = load_rttm("hyp.rttm")["lords-2026-09-16"]
uem = Timeline([Segment(35, 9000)])
for per_side in (0.5, 2.0, 5.0):  # pyannote's collar is the total width
    der = DiarizationErrorRate(collar=2 * per_side, skip_overlap=False)
    print(per_side, der(ref, hyp, uem=uem, detailed=True))

The index: contains Parliamentary information licensed under the Open Parliament Licence v3.0.

2. A scoring protocol: would you be happy for us to report on this from now on?

• (a) Speakers. Correct turn starts to the speech onset where the reference's times aren't onsets (parliamentlive's index); use force-aligned times (AMI, BUT's only_words) as they are. Score pyannote DER with overlap scored over a collar sweep, 0.5 s per side as the headline, plus speakers found / true, one label per member and returning speakers.
• (b) Transcripts. WER against the official record under two normalisation lenses, verbatim speech the record edits out reported apart from errors, with paired bootstrap CIs.

Transcript scoring, in detail

• Frozen lens: Unicode NFKC, casefold, typographic apostrophes folded, punctuation and symbols to spaces. Numbers, spelling and fillers stay as written.
• Normalized lens (the headline): the same fold, then Whisper's EnglishTextNormalizer. It turns spoken numbers, currency and percentages into digits, maps British to American spelling, removes the fillers hmm, mm, mhm, mmm, uh and um, expands contractions and titles ("mr" to "mister"), and drops bracketed text and diacritics.
• Edited records: Hansard and the GLA transcript leave out repeats and false starts. So each inserted word is typed from the words around it: a filler (the list above plus er, erm, ah, eh, uhm, hm, huh), a repetition (the same word beside it), or a restart (a 2–4-word phrase said twice in a row, or a fragment of the next word). Those count as verbatim extras and are left out of content WER, (S + D + other insertions) / N. So "Lords, Lords, Lords" against Hansard's single "Lords" costs content WER nothing. Measured WER keeps every edit, and we report both.
• CIs: paired bootstrap over speaker contributions (units start at each anchored contribution in the record), 10,000 draws, 95% percentile intervals.

We'll publish the protocol as a code-only kit, so you can rebuild the references and run the scoring yourself: builders that turn the parliamentlive index, Hansard and BUT's AMI annotations into timed, speaker-attributed references, plus the DER and WER scorers. Happy to upstream it if you'd like it in the repo.

3. Lords on the protocol

With turn starts corrected to the speech onset (anchors from a Whisper transcript plus Hansard), the same output scores 10.6% at 0.5 s per side (confusion 2.4, missed 8.2), 9.3% at 2 s and 8.4% at 5 s, with 49 of 53 members clean, 14 of 17 returning members with every turn under one label, and 2 merged.

The anchors haven't been checked by ear. If you accept the protocol, we'll ear-check them blind on a frozen sample of 48 turns, weighted towards hand-set and low-confidence anchors, and post the result before scoring anything further against them. The onset-corrected figures become primary only if at least 46 (95%) start within 1.0 s of the onset heard by ear; until then, section 1 is the headline.

4. AMI on the same protocol

Four AMI test meetings (EN2002a, ES2004d, IS1009d, TS3003d: 2 h 29 min, 16 speakers; AMI Consortium, CC BY 4.0) against BUT's only_words reference, whose times come from forced alignment, so nothing needs correcting. Same build, import and final pass, GPU, defaults:

AMI, 4 meetings	Headset mix	Far-field (Array1-01)
DER, 0.25 s per side (AMI's usual collar)	29.2% (confusion 4.8, missed 20.2, false alarm 4.2)	42.6% (16.3, 22.8, 3.5)
DER, 0.5 s per side	23.3% (3.7, 15.8, 3.8)	37.7% (16.2, 18.4, 3.1)
DER, 2 s / 5 s per side	12.8% / 11.6%	27.5% / 20.9%
Speakers found / true	19 / 16	18 / 16
Speakers on a label of their own	16 / 16	13 / 16
Clear replies of 1 s or less on the right speaker	83 of 346 (24%)	63 of 346 (18%)

The hypothesis is the final transcript, one speaker per segment, so untranscribed speech and the second voice in overlaps count as missed. Peak worker VRAM was 3,958 MiB.

Definitions

A reply is clear when at least half of it has nobody else talking, and right when it gets the label of that speaker's long turns (more than 2 s). Speakers found doesn't count UNKNOWN. A speaker is on a label of their own when no other speaker's dominant label is theirs.

5. ASR windows

#449 ships the ASR-window work, so we've dropped our planned PR series; thanks for taking it on. Our full-fixture runs confirm the CPU gain: on Lords, measured WER against Hansard (normalized lens) fell from 48.1% to 23.0% (paired 95% CI on the drop: 18.6–32.9 points). One piece is still worth a look: a backstop that re-decodes speech the decoder skips inside a window. I'll open an issue (#492).

Corrections to earlier posts

• Live labels: the opening post mixed figures from three variants, and I've corrected it in place. The ~400 MiB figure belongs to the 5 s window, the design the live-labels issue (Design: live speaker labels that wait until confident, with the final pass unchanged #487) uses.
• onnxruntime threads: the timings in my 15 Oct reply and the tools README (1.8–2.2× on Parakeet, 3–3.5× on Canary) are superseded by CPU ASR ignores the container's CPU limit: onnxruntime sizes its thread pools to the host #429's controlled test: 2.35× on Parakeet and 3.6× on Canary.
• Ear check (15 Oct reply): 147 of the 151 decisive answers were really spoken, 83 of 83 in VAD-muted Lords audio, and 3 of the 52 decisive catches drew a false "yes". The catches were fake words planted over silence and real words swapped for others, mixed in blind with controls of real transcript words, all 48 of which were heard.

— nullcat (Claudette)

[thebeline]

@Valtora - I noted there were a few bug fixes that you pushed around and after my recent PRs. I am open to feedback on any testing and validation procedures you would recommend me adding to my pipeline to avoid introducing similar bugs in the future. I apologize for adding any additional work to your plate, I'll try to avoid it moving forward.

[Valtora]

Hi @thebeline, not at all, your PRs have been great and any tweaks and bug fixes are well within the expected amount in the natural development of any software. Your work has helped me and the project so please do not take my input on your work as criticism!

Very glad to have your contributions, please do continue the good work and together we'll make sure Nojoin goes from strength to strength.

[thebeline]

Status, so we don't step on each other:

Lords baseline: not posted yet, on purpose. Our first run was on our fork rather than upstream, and not isolated, so I'm redoing it from a clean upstream build before citing any number. The anchored log gives 53 members over 98 turns.

ASR on CPU: setting that up turned up a bigger problem. The default CPU path loses roughly 30% of the Lords sitting, in gaps longer than 20 s that each end at a window boundary. It reproduces on the 2.7.0 and 2.8.0 release images with nothing of ours in the loop. That's Parakeet; Canary fails differently, by looping. It bears on the window setting from #382. The GPU measurement is running now, and the write-up will come with a script you can run against your own stack, plus a proposed fix series.

#426: on it.

Long-meeting notes: #394 covers most of what I'd planned, so I've dropped my version.

Next: the live-labels design issue with Lords numbers, since it touches the same live-lane code as the ASR fix. Timeline notes get built while that's in review.

— nullcat (Claudette)

[Valtora]

Sounds great @thebeline, will leave it with you and check back whenever you're ready.

[thebeline]

The CPU default loses about a third of the Lords sitting, and capping onnx-asr windows at 20 s fixes most of it, on CPU and GPU. The cap also leaves #382's setting with nothing to do.

Shipped behaviour, no harness. I ran the 2.7.0 worker image CPU-only, through the product's prep and transcribe_audio, with no constant changed. Parakeet returns 12,214 words for Lords, where an independent word reference has 20,397. 49 gaps over 20 s hold 31% of the reference words. My sitrep above said each one ends at a window boundary; only 13 do, but those 13 hold half the gap words. 2.8.0 makes the same 40 cuts and matches our benchmark run word for word. That run leaves 36.31% of the reference speech uncovered. The windowing on main hasn't changed since.

Speech lost, %. Lords: reference speech that no output word touches. Seattle and London: VAD speech with no transcript. CPU: int8, 6 pinned cores. GPU: fp32, RTX 4070 Ti 12 GB, image built from upstream 0426af50.

Policy	CPU Lords	CPU Seattle	CPU London	GPU Lords	GPU Seattle	GPU London
Shipped: 240 s CPU, 120 s GPU	36.31	15.07	21.48	6.43	9.76	5.74
120 s, cuts moved back into silence				8.02	8.97	6.05
120 s silence cuts + unlimited backstop	4.18	10.94	11.33	0.56	7.13	4.57
20 s cap	5.79	8.39	8.08	0.88	5.27	3.97
20 s cap + backstop	2.74	7.23	6.79	0.66	5.19	3.54

The cap is your windowing capped at 20 s, with each cut moved back to the latest muted silence, or else to the quietest point. The backstop re-decodes VAD speech that came back with no words, up to 10 s per window. The CPU 20 s cap rows were rescored from the run traces with your last-word rule, which the PR keeps. The cap does almost all of the work. Better cuts alone made GPU Lords worse. Long windows with an unlimited backstop beat 20 s on GPU time, but they still miss Seattle and London on both devices.

What it costs, on GPU, against the shipped 120 s. WER is against Hansard and the GLA transcript (agenda items 3–4 only). "Extra words" counts insertions where both policies have text, plus words in audio the VAD muted. It's in points of reference words, with a limit of +1.

GPU	WER Lords, frozen / normalized	WER London, frozen / normalized	Extra words, normalized, Lords / London	ASR time, Lords	Peak VRAM, Lords
Shipped 120 s	29.29 / 26.03	26.28 / 16.83		36.5 s	4,844 MiB
20 s cap	27.97 / 23.01	29.20 / 18.70	+0.81 / +0.55	44.5 s	3,716 MiB
20 s cap + backstop	27.83 / 22.73	28.32 / 17.65	+0.93 / +0.51	40.6 s	3,712 MiB

• Lenses. Frozen is our pre-registered scorer: lowercase and no punctuation, so "uh" counts, and so does "twenty" against "20". Normalized is Whisper's EnglishTextNormalizer. I made it primary after seeing the frozen results, so it's post-hoc.
• Thin passes. Normalized, cap + backstop passes both checks, but thinly: Lords extra words by 13 of 18,920 words, London WER by 0.18 pp. Frozen, London WER is 2.04 pp worse and extra words fail (+1.96 / +1.08).
• Time. Cap + backstop takes 1.11–1.34× the shipped time on GPU, and within 1.06× on CPU. The shipped and cap-alone GPU times are single runs.
• CPU text, checked by ear. On CPU, cap + backstop counts as failing on extra words (+3.26 / +1.15, normalized). So we ran a blind listening test on a sample of those words, mixed with real-word controls and 54 planted catches (fake words over silence, swapped words). 147 of the 151 judged were really spoken, 83 of 84 in VAD-muted Lords audio, and only 3 of the 54 catches drew a false "yes". The extra words are speech the shipped windows dropped, and the check passes once that's counted.
• Canary, CPU default, first 20 minutes of Lords: 96% WER against a Whisper proxy, with 76% of its 4-grams repeated. With cap + backstop the same slice gives 7.5% WER and 1.9% repeated, and the full sitting gives 5.33% WER. Neither the cap alone nor GPU has been run on it.

#382. Better said now than in a PR: under a 20 s cap, the GPU window setting is a no-op. It accepts 30–240 s, so min() always returns 20. And 20 itself can't be set:

• the API returns a 422 (backend/api/v1/endpoints/settings.py:168-176);
• the floor is 30 s (backend/utils/config_manager.py:88-89);
• a stored 20 is logged as invalid and replaced with 120 s (backend/processing/engines/onnx_asr_engine.py:73-81).

The CPU path, where the big loss is, never reads it (:91-93). #371's aim, a smaller footprint on a shared card, comes with the cap: about 1 GiB less peak VRAM. I'd retire the key in the first PR. If you want the knob, it could become the GPU window itself, with a 10 s floor. Your call.

Checking it yourself. This bundle runs a released worker image CPU-only, through the product's prep and transcribe_audio. It records windows, words, threads and wall time. For each PR, run your main and the PR branch on the three #339 recordings, on one machine, and compare:

• VAD speech lost, which needs no reference;
• Lords reference speech lost (ours: faster-whisper large-v3-turbo word timestamps);
• WER under both lenses, and extra words;
• time and peak VRAM.

Where we disagree, your numbers settle it, and I'll find out why.

Proposed series. Each PR opens after the previous one merges:

1. The 20 s cap on both devices, a single call for inputs up to 30 s, and the feat(asr): make the GPU transcription window an install-wide setting #382 change.
2. A speech map: VAD runs once and engines get its speech spans, so cuts land between speech.
3. The backstop.
4. Which audio copy to decode, only if measurement favours the unmuted copy or a Qwen3 engine needs it.

Does that split suit you, or would you rather have fewer, larger PRs?

Separately, a small one. Nothing upstream sets onnxruntime's thread counts, so CPU sessions run at intra_op_num_threads=0. In a 6-CPU container on a 24-thread host, 2.7.0 took 1.8–1.9× our 6-thread time on Parakeet, and 3× on Canary, and onnxruntime itself logs "Specify the number of threads explicitly" about 80 times per run. 2.7.0's onnxruntime differs (1.20 against our 1.30), but 2.8.0, on 1.31, was slower still (2.2× and 3.5×). I haven't run the one-variable test. Worth its own issue?

— nullcat (Claudette)

[Valtora]

I'm happy with fewer larger PRs and that split works, thank you. The Onnx finding is definitely worth its own issue too.