How should timestamps be handled for audio discarded during decoding?

[dalecurtis]

While trying to fix the negative timestamp test failures in Chromium:

• https://wpt.fyi/results/webcodecs/audioDecoder-codec-specific.https.any.html%3Fopus?label=master&label=experimental&aligned&q=Test%20decoding%20a%20with%20a%20negative%20timestamp

I've run into a question about how things like Opus pre-skip should be handled. The opus test file in this case has a pre-skip of 312 samples (6500us), which Chromium's decoding system happily discards. To ensure timestamps are accurate we adjust the -42 forward (to 6548us).

As I see it, we have a couple choices:

• AudioDecoder shouldn't discard anything.
• AudioDecoder shouldn't adjust timestamps for discarded audio (not sure how this works when full packets can end up discarded).
• Agree that timestamps should be adjusted for discarded audio.
• Do nothing and switch to a Opus test file with zero pre-skip

@padenot @youennf @tguilbert-google WDYT?

[dalecurtis]

This is some what related to #626 and I could have sworn Paul and I chatted about this before, but I couldn't find anything in my e-mail.

[Vanilagy]

How does FFmpeg do it?

I'm leaning towards "Agree that timestamps should be adjusted for discarded audio", although it depends on the semantics intended by the codec.

[dalecurtis]

@Vanilagy ffmpeg is a bit inconsistent, for some types of discard (generally container marked), you can configure the codec to either discard or emit the data via AV_CODEC_FLAG2_SKIP_MANUAL. For AAC and MP3 there appear to be cases where ffmpeg chooses to discard regardless of the flag though (presumably bitstream marked).

As ffmpeg does, another solution would be to add an option to control discard, but it's hard to see the value in that.

[dalecurtis]

Per discussion in the working group today the consensus is to discard automatically and adjust timestamps forwards. I've prepared a PR to codify this in the spec text. @padenot will prepare a more complete answer for encoding priming on #626 . I'll also prep a PR after this one lands to introduce the end trimming inherent in opus.