Skip to content

Keep non-ASCII letters and punctuation in ids derived from names - #1605

Merged
josephjclark merged 5 commits into
release/nextfrom
fix-slugify-unicode
Nov 2, 2026
Merged

josephjclark merged 5 commits into
release/nextfrom
fix-slugify-unicode

Conversation

@midigofrank

@midigofrank midigofrank commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Short Description

Workflow, step and channel ids now keep non-ASCII letters and punctuation from their names, so they match the ids Lightning uses. Channels are also tracked by their Lightning id, so a change to their keys can't delete them on deploy.

Fixes #1602
Fixes #1603

Background

Until OpenFn/lightning#5106 (September 2026), Lightning only allowed ASCII letters, digits, spaces, _ and - in job and workflow names, and slugify was written for that. Lightning now accepts any character except control characters, so names like 患者確認 or Event-based workflow 1 (rename) reach the CLI. Lightning's own v2 export builds ids by lowercasing the name and replacing spaces with -, and nothing else. This PR follows the same rule.

Implementation Details

This PR has two commits, each with its own changeset.

Ids derived from names

slugify used to turn every non-word character into a hyphen. Names in other scripts lost their letters (患者確認 became an empty id), and names with punctuation got ids that didn't match Lightning's (Event-based workflow 1 rename and Event-based workflow 1 (rename) both became event-based-workflow-1-rename). It now trims, NFC-normalizes and lowercases the name, and turns spaces into - as Lightning does. Characters that aren't safe in file names (/ \ : * ? " < > |) and a leading . are also replaced with -. Those are the only cases where our ids differ from Lightning's.

The same slugify is used for workflow, step, channel and project ids. Project ids for names with punctuation change too. That's harmless, because projects are matched by uuid first.

Ids are no longer URL-safe. Nothing puts a local id in a URL: they're only used as file and folder names and as YAML keys.

Some different names still resolve to the same id:

  • names that differ only by case or by space vs hyphen (Foo Bar and foo-bar). Lightning's export rejects these too.
  • names that differ only by a replaced character (Q3/Q4 and Q3:Q4) or by a leading dot (.foo and -foo). Lightning keeps these characters in its ids, so its export allows these pairs.

A pull now fails with an error listing the clashing workflow, step or channel names.

Channels

Channels in resources.yaml have no ids, and merge matched them to Lightning's channels by key only. Any remote channel whose key wasn't found locally was sent to Lightning with delete: true. That caused two problems:

  • A changed key deleted the channel. The new slugify changes some channel keys. A hand-edited key in resources.yaml has the same effect. Either way, the channel lost its id on the next deploy and was deleted and recreated, which also removes its data in Lightning.
  • Clashing keys updated the wrong channel. When two channel names clashed (My Channel and my-channel), toResourceChannels wrote the second one under a -2 suffix. That only got the pull through. On deploy, both remote channels had the key my-channel, so merge matched only one of them: local my-channel might update the other channel, and local my-channel-2 was created as a new channel.

The fix tracks channels the way workflows are tracked:

  • The checkout file now records channel_uuids by key, as it already does for workflow_uuids, and from-fs reads them back.
  • mergeChannels matches by id first, then key, then name. Matching by name covers checkouts written by an older CLI, which have no channel_uuids.
  • A remote channel is only flagged as removed if its id isn't on any merged channel.
  • The -2 suffix is gone. Clashing channel names are rejected on pull like workflows and steps, so a project with channels named My Channel and my-channel that used to pull will now fail until one of them is renamed.

One case can still recreate a channel: the checkout has no channel_uuids, the channel's name has punctuation, and the name was changed locally before deploying. The first pull with this version writes channel_uuids and closes that gap.

Known limits

Windows reserved names (CON, NUL) and trailing dots aren't handled.

QA Notes

  • Pull a project with workflow, step or channel names containing non-ASCII characters or punctuation, e.g. Flujo de atención (v2.0). The folder and step ids should keep those characters.
  • On a project pulled with an older CLI, pull again. Workflow folders and resources.yaml keys for those names will change. Channels should keep their ids and should not show up as removed on deploy.
  • Pull with an older CLI. Edit a step in a workflow whose step names have accents or punctuation, then deploy with this CLI without pulling first. The step should keep its uuid and should not be recreated.
  • Pull with this CLI, change a channel's key in resources.yaml, then deploy. The channel should be updated in place, not deleted and recreated.
  • Give two workflows (or two steps in one workflow, or two channels) names like My Flow and my-flow, then pull. It should fail with an error naming both.

AI Usage

Please disclose whether you've used AI anywhere in this PR (it's cool, we just
want to know!):

  • I have used Claude Code
  • I have used another model
  • I have not used AI

You can read more details in our
Responsible AI Policy

slugify replaced every non-word character with a hyphen, so names in
other scripts lost their letters and their ids didn't match Lightning's.
It now only replaces characters that aren't safe in file names.

Names that differ only by case or by space vs hyphen still give the same
id. A pull now rejects them, as Lightning's export does, so the
resources.yaml writer no longer adds -2 suffixes to channel keys.
The new slug rules change some resources.yaml keys, and merging only
matched channels by key, so a channel whose key changed lost its id and
was treated as removed. The checkout file now records channel_uuids by
key, as it already does for workflows. A merge matches channels by id
first, then by key, then by name.
@github-project-automation github-project-automation Bot moved this to New Issues in Core Oct 29, 2026
@midigofrank midigofrank changed the title Fix slugify unicode Keep non-ASCII letters and punctuation in project ids Oct 29, 2026
@midigofrank midigofrank changed the title Keep non-ASCII letters and punctuation in project ids Keep non-ASCII letters and punctuation in ids derived from names Oct 29, 2026
@midigofrank
midigofrank marked this pull request as ready for review October 29, 2026 15:30
@midigofrank midigofrank self-assigned this Oct 29, 2026
@josephjclark

Copy link
Copy Markdown
Collaborator

@midigofrank can you add a separate changeset for the channels change? We'll keep the work here, I just want it represented in the release notes (and the PR body)

@josephjclark

Copy link
Copy Markdown
Collaborator

This was sort of done here: #1531 - but I'm going to prefer this PR (and extend it)

Also there was a discussion around this when looking at support for apollo OpenFn/lightning#4577

I can't find the thread now - but it was anticipated that some step names would clash (thumbsup/thumbsdown emoji was the example). I'll accept the fixes but I suspect this is the sort of thing that comes out of QA and doesn't impact the real world

@josephjclark
josephjclark changed the base branch from main to release/next November 2, 2026 15:29
@josephjclark
josephjclark merged commit 82a0cae into release/next Nov 2, 2026
11 checks passed
@josephjclark
josephjclark deleted the fix-slugify-unicode branch November 2, 2026 15:29
@github-project-automation github-project-automation Bot moved this from New Issues to Done in Core Nov 2, 2026
josephjclark added a commit that referenced this pull request Nov 3, 2026
* Keep non-ASCII letters and punctuation in ids derived from names (#1605)

* Keep non-ASCII letters and punctuation in ids

slugify replaced every non-word character with a hyphen, so names in
other scripts lost their letters and their ids didn't match Lightning's.
It now only replaces characters that aren't safe in file names.

Names that differ only by case or by space vs hyphen still give the same
id. A pull now rejects them, as Lightning's export does, so the
resources.yaml writer no longer adds -2 suffixes to channel keys.

* Track channel ids in the checkout file

The new slug rules change some resources.yaml keys, and merging only
matched channels by key, so a channel whose key changed lost its id and
was treated as removed. The checkout file now records channel_uuids by
key, as it already does for workflows. A merge matches channels by id
first, then by key, then by name.

* add changeset for channels change

* comments and tests

---------

Co-authored-by: Joe Clark <jclark@openfn.org>

* Fix pull alias (#1610)

* add failing tests

* fix

* changeset

* versions

---------

Co-authored-by: Midigo Frank <39288959+midigofrank@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

2 participants