Figure out a consistent story for translating theme content

This would basically be something that helps theme authors provide + maintain translations for their theme.

It will be a combination of documentation and some build-time logic for managing + validating the translations, as part of generating the distributions.

Just a note that we are currently being bitten by this in the PyData Sphinx Theme.

We've added a folder with locale information, and have just the .po files checked into git history. However because of the way pybabel works it is not trivial to put the compiled .mo files into a different directory than their sister .po file. This means that we keep the .mo files in the same folder (as pybabel does by default).

However, this means that we can't use stb serve because the .mo files are compiled at build time, and don't live in compiled_assets (ref: #18) and so it detects them changing each time. I also don't think stb serve has the ability to ignore extra folders the users provides (which would be useful).

Do you have links for me to look at, to understand how pydata-sphinx-theme manages translations?

Yep - links below:

Here's our locales folder: https://github.com/pydata/pydata-sphinx-theme/tree/main/src/pydata_sphinx_theme/locale
Here's the lines in our webpack config where we compile translations at build time

https://github.com/pydata/pydata-sphinx-theme/blob/157c9ab2c93e755141e48d13b5f193c6169d9dd8/webpack.config.js#L21-L26
Here's all our docs on translations: https://pydata-sphinx-theme.readthedocs.io/en/latest/community/topics/i18n.html

If stb compile also ran the translation logic, instead of webpack, would that work for y’all?

Yeah I'd much rather rely on a stb functionality for this rather than hand-rolling webpack commands. Especially if it followed recommend practices for pybabel do I didn't have to figure out if I'm doing it right out of the box haha. It'd also be great if we could have the .po files in assets/locale and the .mo files in static/locale but i couldn't figure out a non hacky way to do this.

I'm not an expert but AFAICT we are just doing standard pybabel commands largely based on the docs default recommendations

Neato! Depending on how my recovery from a medical procedure I had today goes... I might take a stab at this today. Otherwise, please poke me on some other channel about this coz as it'll definitely leave the front of my mind in a day or two. 🙈

@pradyunsg

Finally, I'm circling back here! I recently spent much time refactoring our translation workflows and then patching the packaging in PST, which is good because it helped me think about how to include this in stb.

💡 Ideas and notes:

We have had resources to call the translation compilation from within Webpack due to how stb package currently works and how generated artefacts are found and added to the distribution files (everything is expected to be in src/theme/<my-awesome-theme>/static/ and adding any extras to the pyproject.toml but still within the src/theme/<my-awesome-theme>/static/ directory)
The above has proved tricky for development and maintenance since:

It creates an infinite loop for us when trying to work on the theme and docs as we end up compiling JS artefacts, processing HTML templates, compiling po files (even using the re-ignore flag 🤔, though it should not, I believe).
It also obscures some of the translation maintenance tasks; since the po files are compiled through webpack maintainers/contributors often forget to update the catalogue pot files, leading to outdated/incorrect translation files (it would be great to include some check here too to identify when these files are out of date and send a warning or the likes)
It would be better for maintainers to be able to extract, update, and compile the translation files from within stb and specify the location of the po/pot files as well as the target location of the compiled mo files independently from the JS/scss/css assets (this would be quite helpful for development and we can provide a good default of static/locale like Chris suggests before, while allowing users also to customise if needed, and integrate this into the stb package to ensure localisation/translation files are correctly updated and included for distribution)

There should be an individual stb command to process the translation files (and as mentioned above, later called by stb package to streamline the packaging/distribution workflow), but that is not necessarily tied to stb compile directly or as a hack like we currently do in PST due to the lack of a better alternative.

I can work on a prototype for this now that I better understand the involved pieces and the requirements from a maintenance POV WDYT @pradyunsg?
If so, I would consider adding a stb translate/internationalize command and take it from there.

@pradyunsg

emerges from a dark place, squinting at the brightly lit outdoors
checks date

Happy New Year!

It creates an infinite loop for us when trying to work on the theme and docs as we end up compiling JS artefacts, processing HTML templates, compiling po files (even using the re-ignore flag 🤔, though it should not, I believe).

I'm gonna guess that this is because sphinx-autobuild has changed something underneath this project... I'm gonna resist the urge to dive into figuring this out right now and finish writing this response (which I've been meaning to do for a few weeks now).

I can work on a prototype for this now that I better understand the involved pieces and the requirements from a maintenance POV WDYT @pradyunsg?

This would be great! 🎉

later called by stb package to streamline the packaging/distribution workflow

I don't think stb package is the right layer to be "doing" all of this (any build logic should be in the build-backend), but translation-related behaviour should definitely be exposed via the stb package command.

we can provide a good default of static/locale like Chris suggests before

One concern I have with placing any translation-related in the static/ directory is that Sphinx would end up picking up these files and shoving them in the output for end-user's documentation sets. It would be better to store these outside of the src/[name]/theme/ folder (which is the base path that themes provide to Sphinx, for where to attempt to look for the theme's templates/static files).

IIUC, these files should not end up in the documentation output, and *.{mo,po,pot} should all end up being installed on the documentation author's machine.

This comment is a bit of a data/thought dump, as a part of my process for (re)loading context about this issue in my head. I'm putting it here for visibility + posterity. Apologies for the size of this text wall! 🙈

If someone ends up actually reading this (please don't feel like you need to!)... let me know if I've misunderstood anything, or if there's more options than the ones I came up with below -- I'm not particularly strongly attached to these ideas. I think of this as an unshaped blob of clay, it still needs to be moulded into a good $thing!

If you see something that can be better, I'd be happy to know!

In terms of what stb can help with... let's see... what does a theme need to do after all the bootstrapping work is done (those steps are documented by Sphinx already + is more of an "initialize" concern which is WIP -- #34):

Add a new language to translate into.
Ensure that .po files & .pot are up to date as-committed.
Ensure that .mo files are generated when serving documentation, when publishing the package and are not committed.

That's... yea, that's a reasonably tractable amount of stuff. There's definitely a bunch of different operations!

A nested bullet list because my thoughts are all over the place...

Useful references:
What is the theme's locale's name?
- Match the [project] -> name I think, since that's already required to be unique to the project on PyPI and that's also aligns with Sphinx's guidance on the topic...
1. Choose a name for your message catalog, which must be unique. Usually the name of your extension is used for the name of message catalog.
- pydata-sphinx-theme reuses the sphinx name. I guess that works too. 🤔
Let's assume that we'd use src/[name]/locales to store all of .mo, .po and .pot files for now.

It's likely worthwhile to enforce a specific directory as the localedir for the project, for consistency with how stb handles assets (it's one-less-decision for the theme authors), it'll maybe be a simplifying assumption as I work through the rest of this (eg: adding an equivalent ignore to assets/).
If we store a list of translation "destination" languages in pyproject.toml, what can stb do with that?
- Adding a language (1) would make sense as a command that modifies the list + initializes a new .po file.
- The compile process can be a loop over languages, instead of a loop over the file system.
  - It would probably still be necessary to add a validation check to ensure that a file isn't missing / a file isn't extra.
- This will be an easy-to-audit location to check for supported languages of a theme (but the src/[name]/locales directory would also work even-better for this!).
- Unclear if this is useful, based on a first order exploration of the idea.
If there's something here that can slow the development cycle, can we skip work?
- Not re-generating a .mo if it is newer than the corresponding .po (modelling the .py -> .pyc relationship) -- within the compile command, but not in the wheel build.
stb compile: Any changes?
- Option 1: Nothing changes.
- Option 2: Try to shove all of translation into stb compile.
  - Regenerate the assets by running npm run-script build.
  - 🆕 Update the .pot & .po files. (2)
  - 🆕 Generate the .mo files. (3)
  - dealbreaker: no clean way to add a language outside of a separate command. :/
- Option 3: Add options for the two types of assets being built.
  - stb compile --exclude-assets would do the [sources] -> .pot -> .po -> .mo stuff. (2) (3)
  - stb compile --exclude-locale would do what it does today.
  - dealbreaker: no clean way to add a language outside of a separate command. :/
- Option 4: Move things around, with a new stb assets compile along side a stb locale compile -- with stb compile running them both.
  - stb locale [...] would do the [sources] -> .pot -> .po -> .mo stuff. (2) (3)
  - stb assets compile would do what it does today.
  - stb locale add [...] can handle adding languages. (1)
stb serve: Any changes?
- Option 1: Ignore everything in the localedir.
  - Should be easy-enough (I've jinxed this, haven't I?).
- Option 2: "Smart" rebuilds that's a nicer experience.
  - Rebuild site when the .po/.pot file is updated "out of build" (i.e. manually).
  - Rebuild site when the project's HTML/.py etc are changed.
  - Rebuild .po/.pot when the project's HTML/.py etc are changed.
  - This likely needs some control over sphinx-autobuild's rebuilds, since we want conditional rebuilds here.
  - IDK if this is possible -- this is definitely a nicer experience though for someone using a local workflow for translating the text, because this can give them an end-to-end preview of their changes' effects.
stb package: No changes, continue calling the build-backend via python -m build.
Build backend:
- sdist changes: None?
  - .mo files must be excluded via git, so file inclusion doesn't change.
  - Maybe... validate that .po / .pot files are up to date?
- wheel changes: Handle (3) above.
  - Compile .po files into .mo files, before generating the final bundle.
Do we care about multi-repo setups for handling translations? (eg: Python's setup uses a bunch of separate repositories: https://github.com/python/?q=python-docs-&type=all&language=&sort=)
- Not now -- let's cross that bridge when we get there.
CI: Do we care?
- The check would be whether the .po / .pot files are up to date. Yea... that seems worthwhile and is basically what step 2 is.
- We can't do this as a part of stb compile alone, since that needs to modify these files as an expected pattern of use.
- Maybe also that the .mo files are not committed.
- An stb ci might be a good idea!
- Oh oh! We can check for the other folders as well here! Oooh... this might be a good idea in general!

What this could look like?

This is an attempt to visualize a possible outcome here, to see if I still need more data here...

The entire translation-related logic is conditional behind a "does a stc/[name]/locales folder exist?" check (which helps maintain backwards compatibility with existing themes that don't have translations).

stb compile [--exclude-assets] [--exclude-locale]
- Expected to only modify files in src/[name]/theme/[name]/static.
- Runs npm run-script build.
- 🆕 Runs pybabel-based extract + update + compile.
stb package
- A very thin wrapper around python -m build.
  - Triggers sdist build.
    - Bundles files from the tree.
  - Triggers wheel build.
    - Runs npm run-script build.
    - 🆕 Runs pybabel-based extract + update + compile.
🆕 stb assets compile: A thin wrapper around npm run-script build.
🆕 stb locale add: A thin wrapper around pybabel init [...].
🆕 stb locale compile: Runs pybabel-based extract + update + compile.
Maybe sometime in the future:
- 🆕 stb locale ci: Verify that extract + update doesn't modify files.
- 🆕 stb assets ci: Ensure that compilation doesn't touch assets outside of assets/.
- 🆕 stb ci: Mirrors stb compile but for ci subcommand.

The main question around doing all the pybabel's extract + update + compile steps as a part of stb compile is whether there's any short-circuiting / caching in that process in pybabel. If not... that's an argument against this design. In that case...

stb compile
- Expected to only modify files in src/[name]/theme/[name]/static.
- Runs npm run-script build.
- 🆕 If source files changed... run pybabel extract.
- 🆕 If .pot file changed... run pybabel update.
- 🆕 If .po file changed... run pybabel compile.
stb package
- A very thin wrapper around python -m build.
  - Triggers sdist build.
    - Bundles files from the tree.
  - Triggers wheel build.
    - Runs npm run-script build.
    - 🆕 Runs pybabel-based extract + update + compile.
🆕 stb locale add <lang>: Run pybabel init [...] <lang> [...].
🆕 stb locale extract [--check]: If source files changed... run pybabel extract.
🆕 stb locale update [--check]: If .pot file changed... run pybabel update.
🆕 stb locale compile [--check]: If .po file changed... run pybabel compile.
🆕 stb ci: Runs all the logic that has a check mode.

The --check options would ignore the condition and check if any output file was modified.

@pradyunsg

emerges from a dark place, squinting at the brightly lit outdoors
checks date

Welcome back @pradyunsg. Thanks for the writeups (which I only skimmed through and need to digest).
I will go through these and can start working on a POC soon-ish (?), I am sure I will have to come back here and/or add questions as I work on that but it should be fine.

OK, so, I've fallen down a rabbit hole today.

First off, I no longer think that .pot needs to be committed. I'm not gonna enforce omission but it does feel a bit redundant + conflict-prone to have committed, since it's generated from the source files directly and will have churn as files change.

I looked at the top 10 on https://sphinx-themes.org/ (+ python-docs-theme) and found that we (as a community) are doing something subtly different in every theme. :)

Locale/translation management

Theme	Domain	Babel	Translation Platforms
sphinx-rtd-theme	sphinx	babel.cfg + pybabel's setuptools commands	Transifex
pydata-sphinx-theme	sphinx	babel.cfg + pybabel's CLI	Transifex
shibuya	sphinx	babel.cfg + pybabel's CLI	None
sphinx-book-theme	booktheme	NO -- custom script	None
python-docs-theme	python-docs-theme w/ fallback to sphinx	.babel.cfg + babel_runner.py wrapping pybabel's CLI	Transifex
(proposal here)	`[project].name`	babel.cfg (probably)	Transifex compatible

Packaging

Theme	Repository	Source distribution	Wheel distribution	`.pot` generation	`.po` generation	`.mo` generation	When .mo is generated
sphinx-rtd-theme	`.pot` `.po` `.mo`	`.pot` `.po` `.mo`	`.pot` `.po` `.mo`	Babel `extract_messages`	custom `UpdateTranslationsCommand`	Babel `compile_catalog`	Development
pydata-sphinx-theme	`.pot` `.po`	`.pot` `.po`	`.pot` `.po` `.mo`	`pybabel extract`	`pybabel update`	`pybabel compile`	Wheel build
shibuya	`.pot` `.po` `.mo`	`.mo`	`.mo`	`pybabel extract`	`pybabel update`	`pybabel compile`	Development
sphinx-book-theme	JSON	JSON	JSON `.po` `.mo`	none	custom `_compile_translations.py`	custom `_compile_translations.py`	Wheel build
python-docs-theme	`.po`	`.po` `.mo`	`.po` `.mo`	`pybabel extract`	`pybabel update`	`pybabel compile`	CI, before sdist
(proposal here)	`.po`	`.po`	`.pot` `.po` `.mo`	`pybabel extract` (only for development)	`pybabel update`	`pybabel compile`	Wheel build

This is very interesting, just a few things I've noticed about python-docs-theme:

We don't use pybabel directly, like sphinx-book-theme we use our babel_runner.py script.
We don't quite use TX, there's no .tx folder in the repository, and we accept PRs.
We are unnecessarily adding the PO files to wheels :-/
The theme falls back to Sphinx as we add translations there in CPython.

Wrt the domain fallback: I think the python-docs-theme fallback uses Sphinx's own translations, e.g. from themes/basic directory. What are counterarguments to include the fallback to sphinx domain in in the proposed translation story? I think it should prevent confusion or help when customizing or forking from built-in templates.

Wrt source distibution: wouldn't it be worth to include .mo files in source distributions to speed up building the theme package for cases of not covered platforms with pre-built wheels? Or would such optimisation be too much churn/size?

Wrt source distibution: wouldn't it be worth to include .mo files in source distributions to speed up building the theme package for cases of not covered platforms with pre-built wheels? Or would such optimisation be too much churn/size?

Indeed, I wrote python/python-docs-theme#312 which would do that.

Wrt source distibution: wouldn't it be worth to include .mo files in source distributions to speed up building the theme package for cases of not covered platforms with pre-built wheels? Or would such optimisation be too much churn/size?

The wheel for themes is pure-Python so there's really no platform where you can use Python and not a built wheel.

We're basically putting a compiled artifact in the source distribution. It's not banned or a problem, but it's also not necessary IMO.

What are counterarguments to include the fallback to sphinx domain in in the proposed translation story?

See "What is the theme's locale's name?" above. I don't know enough yet to have an opinion on this.

What are counterarguments to include the fallback to sphinx domain in in the proposed translation story?

See "What is the theme's locale's name?" above. I don't know enough yet to have an opinion on this.

A side effect of reusing the sphinx domain in a theme is that it will reuse core Sphinx translations, i.e. for a string marked for translation that's already used in core Sphinx (basic theme), it will appear already translated in a fresh translation. When using theme's own domain, a per-message fallback "hook" (like in python-docs-theme) is needed to achieve similar effect.

PS IMHO it's a missing feature of Python's gettext, the Python's gettext's "native" fallback doesn't work this way and such hooks are needed, iirc something similar is used in Django.

The wheel for themes is pure-Python so there's really no platform where you can use Python and not a built wheel.

We're basically putting a compiled artifact in the source distribution. It's not banned or a problem, but it's also not necessary IMO.

In this case it somewhat is necessary in practice, there is currently no build hook to compile the translations (the PR does add one). They have to be manually compiled before building the wheel.

Navigation Menu

Repository navigation

Figure out a consistent story for translating theme content #24

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Figure out a consistent story for translating theme content #24

Description

Activity

choldgraf commented on Feb 25, 2023

pradyunsg commented on Feb 25, 2023

choldgraf commented on Feb 25, 2023

pradyunsg commented on Feb 25, 2023

choldgraf commented on Feb 25, 2023

pradyunsg commented on Feb 25, 2023

trallard commented on Nov 25, 2024

pradyunsg commented on Jan 2, 2025

pradyunsg commented on Jan 2, 2025

trallard commented on Jan 3, 2025

pradyunsg commented on May 25, 2026

StanFromIreland commented on May 26, 2026

m-aciek commented on May 27, 2026

StanFromIreland commented on May 27, 2026

pradyunsg commented on May 27, 2026

m-aciek commented on May 27, 2026

StanFromIreland commented on May 27, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions