Emotive speech markup

[Sciumo]

A Markdown extension for emotive speech could be better than SSML in expressiveness and ML training efficiency because of its simplicity and tokenizer friendly syntax.
Markdown's straightforward annotations make it easier to read and edit than XML like markup language.

[fakerybakery]

That's a cool idea! How would it be integrated into the phonemizer?

[Sciumo]

there would need to be an extension to the tokens to encode breath and tone variance.
ingressive[1] and egressive [2] breath control and emphasis.

extended characters for
strong
soft
whisper
yell

so the Markdown encoding would then provide emotive post processing.

Possible:
Hey (capitalized sentence emphasis and timing break)
Hey! (exclaim interest sentence)
Hey! (bold ~ strong emote interest)
hey (italic ~ quizzical interest)
(hey) (italic paren ~ whisper)
HEY (cap ~ yelling)
HEY! (bold cap ~ angry yelling)
: hey this is a sentence. (definition ~ narrative voice)

[1] https://en.wikipedia.org/wiki/Ingressive_sound
[2] https://en.wikipedia.org/wiki/Egressive_sound

[fakerybakery]

Cool! So these tokens would be added to the phonemizer?

[Sciumo]

need to review literature for existing references [1] including tonality [2]

I see the workflow like this.

1. enumerate initial set definitions with IPA references
2. assign tokens to definitions
3. extend token dictionary
4. map markdown to token phoneme methodology
5. build training dataset with emotive markdown [3,4]
6. verify markdown to phoneme translation
7. fine tune base model with dataset

[1] https://en.wikipedia.org/wiki/Phonological_hierarchy
[2] https://en.wikipedia.org/wiki/Tone_terracing
[3] https://zenodo.org/records/1188976
[4] https://www.youtube.com/watch?v=Y7OQoNEu3dY

[fakerybakery]

Cool! Do you have any ideas on how the dataset would be created?

[Sciumo]

I sent an email to maintainers of RAVDESS dataset, which would be the training dataset.

1. Run the dataset through STT, create JSONL dataset.
2. Write a script to do first class markdown and canonical phoneme translation
3. Review by hand and edit the JSONL, probably with an editing script.
   ...
   X) Publish

[1] https://psychlabs.torontomu.ca/smartlab/resources/speech-song-database-ravdess/