Repository navigation
fix: keep WebVTT cue text around inline markup - #353
Open
Kevin Turcios (KRRT7) wants to merge 4 commits into
Open
Kevin Turcios (KRRT7) wants to merge 4 commits into
Kevin Turcios (KRRT7) wants to merge 4 commits into
Conversation
parse_voice_tags captured each voice segment only up to the first "<", so
inline cue markup truncated the text: "<v Alice>I <i>really</i> mean it"
was ingested as "I". Cues without a voice tag kept their markup verbatim,
voice tags with classes ("<v.loud Alice>") were not recognized, and text
before the first voice tag was dropped. Split segments at voice tags and
strip the remaining cue markup while keeping its text.
Co-authored-by: wtfyd-agent[bot] <328989958+wtfyd-agent[bot]@users.noreply.github.com>
The class part of the voice-tag pattern also matched ".", so an unclosed "<v.!.!.!..." could be split into classes in exponentially many ways (CodeQL "Inefficient regular expression"). Exclude "." from class names and require the speaker to start with a non-space so each input splits only one way. Co-authored-by: wtfyd-agent[bot] <328989958+wtfyd-agent[bot]@users.noreply.github.com>
Bernhard Merkle (bmerkle)
approved these changes
Oct 18, 2026
Bernhard Merkle (bmerkle)
left a comment
Collaborator
There was a problem hiding this comment.
LGTM — verified locally.
- The new tests: 7 cases fail against
main'stranscript_ingest.pyand pass with the fix; fullmakeis green (820 passed). - Differential against
webvtt: for all 908 cues intests/testdata/*.vttthe joined segment text equalscaption.text(whitespace-normalized), and the same holds for<v>without annotation,<V>, multi-word speakers, tab afterv, multi-line cues, and<lang>/<ruby>spans. - The second commit's
_VOICE_TAG_PATTERNis linear; CodeQL alert #43 shows as fixed.
Non-blocking suggestions:
_CUE_TAG_PATTERN = r"<[^>]*>"is quadratic on cue text with many unclosed<(each<scans to the end): about 0.09 s at 20k chars and 2 s at 100k. Not realistic for well-formed VTT, but transcripts are external input.r"<[^<>]*>"makes it linear and strips identically on everything above.- Text after
</v>and before the next voice tag is attributed to the preceding speaker:<v Alice>a</v> mid <v Bob>b</v>→('Alice', 'a mid'). Strictly it's outside any voice span, so(None, 'mid')would be more faithful, and consistent with how this PR now treats text before the first voice tag. Rare in practice; fine as a follow-up or a docstring note.
(FYI, unchanged by this PR: character references such as &/< stay undecoded, same as webvtt's caption.text; html.unescape after stripping tags would be safe if you ever want plain text in the index.)
Contributor
Author
|
Thanks! Took suggestion 1: I'll leave the text-after- |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
parse_voice_tags(used byingest_vtt_transcriptandget_transcript_speakers) lost or garbled caption text whenever a cue contained WebVTT markup:raw_text<v Alice>I <i>really</i> mean it</v>('Alice', 'I')('Alice', 'I really mean it')<v Bob>Use the <c.code>make</c> command('Bob', 'Use the')('Bob', 'Use the make command')I <i>really</i> mean it(no voice tag)(None, 'I <i>really</i> mean it')(None, 'I really mean it')<v.loud Alice>Hi <b>there</b></v>(None, '<v.loud Alice>Hi <b>there</b></v>')('Alice', 'Hi there')Intro <v Bob>hello</v>('Bob', 'hello')(None, 'Intro'), ('Bob', 'hello')</v>,<i>,<b>,<c.class>, timestamp tags) is stripped while its text is kept, matching whatwebvtt'scaption.textdoes.<v.class Speaker>) are recognized, aswebvtt's own voice pattern allows.tests/testdata/*.vtt.Validation
parse_voice_tagstests (the table above, plus timestamp tags and a plain multi-voice regression case), and a test that parses a real.vttfile throughwebvtt.read. All markup cases fail without the fix.