Skip to content

Feat/japanese romanization - #25

Open
PeekMusic wants to merge 2 commits into
Ryoku-dev:mainfrom
PeekMusic:feat/japanese-romanization
Open

PeekMusic wants to merge 2 commits into
Ryoku-dev:mainfrom
PeekMusic:feat/japanese-romanization

Conversation

@PeekMusic

Copy link
Copy Markdown

Follow-up to #23: Japanese wasn't part of the lyrics romanization yet. It needs a dictionary, and #23 only left the size question open (measurements, reminder after the merge). This adds it with vibrato + mecab-ipadic, the option with good readings, while keeping the dictionary out of the binary and out of resident memory.

What changed

  • Japanese readings (crates/core/src/romanize/japanese.rs, new): lines are tokenized with vibrato and mecab-ipadic, and each word's dictionary reading is turned into Hepburn-style romaji. Words are joined by grammar the way romaji lyrics are usually written, particles get their spoken reading, and a small curated list corrects IPADIC misreadings seen in real songs.
  • Detection (crates/core/src/romanize.rs): Japanese songs, which Feat/lyrics romanization #23 skipped, are now read with the dictionary. Only that path runs on the blocking pool, and if it fails the lyrics are returned without a reading rather than dropped.
  • Memory: the dictionary is loaded only for a Japanese song and released right after, with the heap trimmed, so it is never kept resident. Readings are cached per song, and a corrupt dictionary is not retried. Without the dictionary file, Japanese gets no reading, as before.
  • Packaging (packaging/arch/PKGBUILD): the dictionary is shipped as package data instead of being embedded in the binary. makepkg fetches it with a checksum, the package installs it under /usr/share/ryotunes/dict/ with its license, and nothing is downloaded at runtime.
  • Dependencies: vibrato without its training feature. Decompression reuses the workspace's existing ruzstd.
  • Copy: the setting's description in both clients and the French translation now lists Japanese.

Why

Japanese was the one major script #23 left without a reading, and its kanji can only be read with a dictionary. Local readings keep it offline and provider-independent, like the rest of #23. Provider transliterations (Boidu's TTML, Netease's romalrc) turned out too patchy to rely on (details).

Surface area

lyrics (ryotunes-core romanization, get_lyrics in the daemon and the Tauri host), packaging (PKGBUILD source, installed data, license), settings copy (QML, Svelte, fr.json)

Validation

  • ./scripts/release-check.sh
  • cd ui && pnpm check && pnpm build
  • cargo fmt --all -- --check
  • relevant Rust/frontend tests: 18 romanize tests pass. See below.
  • affected Ryoku/Hyprland lifecycle tested when applicable: native QML client against a locally built ryotunesd with the dictionary, on Ryoku/Hyprland. Several Japanese songs, a cold start with an empty cache, and the toggle.
  • makepkg --verifysource downloads the dictionary and passes the checksum.

On Linux ryotunes only launches the native client, so the Svelte copy change is untested beyond pnpm check.

Regression contracts

  • background WebKit hibernation
  • MPRIS / tray / explicit Quit semantics
  • stable Home DOM / scrolling
  • live Ryoku theme bridge
  • main vs mini-player window rules
  • replacement-package rollback / ryoku-desktop safety: the PKGBUILD gains a source entry and installs two more data/license paths. Package contents and names are otherwise unchanged.
  • steady-state CPU / frontend timer behaviour: no timer and no background work. The dictionary is loaded per Japanese song only and released right after.

Evidence

Size: release ryotunesd 30,374,504 → 30,487,048 bytes (+110 KiB). The package gains the 7.67 MiB dictionary file.

Memory, ryotunesd RSS recorded every 0.1 s for 10 minutes while playing Japanese songs:

RSS
Running, before the first Japanese song ~230 MiB
While the dictionary is loaded (4 songs) 299 / 301 / 315 / 300 MiB
Right after each song 230 / 238 / 249 / 235 MiB
End of the session 233 MiB

No growth across songs; the trim returns the memory.

Time: ~335 ms to load and read a song the first time (release build), ~1.5 µs from the cache. That's on the blocking pool, alongside the lyrics fetch itself. With the lyrics open, a cold Japanese track showed no visible delay.

Readings next to Apple Music's own transliteration (from Boidu's TTML) for 夜に駆ける:

Line This PR Apple
「さよなら」だけだった その一言で全てが分かった 「sayonara」 dake datta sono hitokoto de subete ga wakatta 「sayonara」 dake datta sono hitokoto de subete ga wakatta
日が沈み出した空と君の姿 hi ga shizumidashita sora to kimi no sugata hi ga shizumidashita sora to kimi no sugata
フェンス越しに重なっていた fensu goshi ni kasanatteita fensu goshi ni kasanatteita
僕の心の全てを奪った boku no kokoro no subete o ubatta boku no kokoro no subete o ubatta

Dependencies: Cargo.lock gains vibrato 0.5.3 with bincode 2.0.1, bincode_derive, crawdad, csv-core, unty, virtue and ahash 0.7. None has a security advisory. bincode carries the informational RUSTSEC-2025-0141 ("unmaintained", no patched version), which cargo audit reports as a warning, not a failure. vibrato uses it only to read the dictionary.

Licenses: vibrato is MIT OR Apache-2.0. mecab-ipadic (NAIST) allows redistribution with its copyright notice, which is installed with the package.

Screenshots:
image

Follow-up

  • The IPADIC misreading list is curated from real songs and will never be complete. More cases are one line each in corrected_reading/merge_compounds, plus a test.
  • The v1.2.6 changelog entry for Feat/lyrics romanization #23 mentions Japanese transliterations and Settings ▸ Appearance. With this PR the Japanese part becomes true; the toggle itself lives in Settings ▸ Playback ▸ Engine.

Japanese songs now get a Latin reading too. Kanji have no reading of their own, so the
lines go through vibrato with mecab-ipadic and each word's reading becomes romaji:
auxiliaries and conjunctive te join their verb, the copula stays its own word, the
particles ha/he/wo read wa/e/o, and the kana table covers the extended katakana
loanwords use.

The dictionary (system.dic.zst, 7.7 MiB) is package data, not part of the binary: the
PKGBUILD fetches it with a checksum and installs it under
/usr/share/ryotunes/dict/ipadic/, with its license. ryotunesd loads it only for a
Japanese song, frees it and trims the heap, and keeps the readings per song. Without
the file Japanese gets no reading, as before.
- Romanization never costs the lyrics: a failed or panicking blocking task returns them
  without a reading, and only the Japanese dictionary path goes to the blocking pool.
- A corrupt dictionary is remembered instead of being decompressed for every song; the
  per-song cache keeps the most recently played songs.
- Words join by grammar only, so an interjection's trailing small tsu no longer glues
  the next word (えっ何 → e nani); da right after a verb is the past tense (迷い込んだ →
  mayoikonda).
- Straight quotes alternate open/close, 々〆〇 are not punctuation, full-width letters
  read as ASCII, and a syllabic n before a vowel or y is written n'.
- Misreadings found in real songs are corrected from context: 如何か/如何に, 前世,
  愛して after other text, 手 after a pronoun, and the split compounds 一人/二人 and 苛つく.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant