CLI slugify drops non-ASCII characters from job and workflow identifiers #1388
Description
Came across this while investigating lightning#4577 (allowing special characters in workflow step names). Flagging it here because it's a pre-existing bug in @openfn/project that affects the CLI's pull/deploy round-trip, and it'll get more user-visible once Lightning loosens its own validation.
The slugify() helper in packages/project/src/util/slugify.ts is:
text.replace(/\W/g, ' ').trim().replace(/\s+/g, '-').toLowerCase()
JavaScript's \W is ASCII-only, so any non-Latin character gets treated as a separator. Running it on a few realistic inputs:
Vérifier l'état du patientbecomesv-rifier-l-tat-du-patientcafébecomescafrésumébecomesr-sum- Anything written entirely in a non-Latin script collapses to an empty string
The problem is that slugify() isn't just used for URLs or display. It's used as identity. In parse/from-app-state.ts it sets the local workflow id (line 117), the local job id (line 164), and the keys used for edge references (lines 144 and 187). In serialize/to-app-state.ts it's the workflow key at line 60 and a fallback job key at line 155. So on pull, a non-Latin name becomes a collapsed or empty id locally, and on deploy those collapsed ids flow back as keys and break edge references. Two jobs in the same script would even collide to the same id.
This already affects workflow names today, since Lightning has no charset restriction on workflows. Step names being restricted was the only reason we hadn't seen it more. Once lightning#4577 merges, step names will be free too, and users with non-English names will hit it immediately when they try to pull their project via the CLI.
The fix direction I'd suggest: stop using slugify for identity. Keep the original string or the UUID as the local key. Slugification is still fine for filenames or URLs, but it shouldn't be what decides whether two things are the same thing.
For testing, the golden path is round-tripping a project with non-ASCII names (French accents, Arabic, Chinese, an apostrophe or two) and asserting the deployed state matches what Lightning sent. A worthwhile edge case is two jobs whose names share the same slugified form, to make sure they stay distinct locally.
Related: OpenFn/lightning#4577, OpenFn/apollo#446.
Activity
josephjclark commentedon Apr 22, 2026
It's OK to use a slugified string for identity internally. In fact its great because the user can use the same identifier and there's no conversion.
We just need to use a more robust library - like slugify. So long as it handles non-utf8 characters, we'll get a consistent internal id string and everything is hunky dory.
We should ensure the same standard of slugification as lightning though - that might be the harder bit
josephjclark commentedon Apr 22, 2026
Slugify tests are pretty good, but I cant see any emojiy stuff https://github.com/simov/slugify/blob/master/test/slugify.js
hanna-paasivirta commentedon Aug 27, 2026
A reminder to match Apollo rules so it doesn't wipe out special characters OpenFn/apollo#446
CaptainAni187 commentedon Sep 8, 2026
Opened #1531 for this.
I went with a Unicode-property regex (
\p{L}/\p{N}) rather than the library @josephjclark suggested, because the popular slugify libraries transliterate —cafébecomescafe,патентbecomespatent— which cuts against the point about the user's identifier surviving without conversion, and doesn't help scripts with no Latin transliteration. Easy to swap if you'd rather have transliteration.The thing I was most careful about: since these slugs are identity, a change in ASCII output would silently re-key existing projects. I checked every printable-ASCII pair plus 200k random ASCII strings — 209,025 inputs, zero differences from the current implementation. Only non-ASCII behaviour changes.
One thing to flag: the
uflag needs an ES2018 target andtsconfig.commonsets none, sotscwas defaulting to ES5 and failing with TS1501. I settargeton the project package's tsconfig only, since that file is validation-only. Worth a look in review in case you'd rather handle it differently.I left the Apollo rules alignment @hanna-paasivirta mentioned out of scope.
josephjclark commentedon Sep 9, 2026
Thank you @CaptainAni187 - we're trying to get this resolved quickly so I'll hopefully get back to you today.