Repository navigation
Projects: serialize to new project format (aka state v2) #1080
Description
Currently the CLI and Projects prefers to serialise project files (ie, main@openfn.org.yaml) to a YAML-flavoured state file.
There are a few things that don't work about this state file:
- Loads of app-specific options are just dumped at the top level. There's no structure so it's hard to say what's important
- It's also a bit hard to know what will sync, what's required, and what the CLI needs
This means we need a new format for a Project file - this thing that's a hybrid of project.yaml, state.json and config.json.
We need to consider:
- We need to be able easily distinguish a v2 project file from a v1 state file. Some top level key that tells us
- Project.from('file') will just be given a path to a yaml file. So how does it know if this is v1 state or v2 project?
- What is the name of this when I serialize. Is it
Project.serialize('yaml')? because after all this is like the native Project format. Maybe I can optionally include a version number an argument. Alternatively itsProject.serialize('project')but that seems dumb - The serializer and parser for this needs to be able to migrate (or at least attempt to migrate) across different versions.
- The order should be either fixed or sorted, so that it can be easily diffed. I think we should serialize to a sensible order, but some sub keys like
openfnoroptionsshould be sorted. Workflows should be sorted by name. Step order is tricky because the first step has meaning in a workflow - how do we resolve that? - Does
aliaslive here? It's supposed to be CLI only and not checked in. But it has to sit somewhere right? I guess it's checked in but not pushed to the app
Ideas and structures:
id:
name:
meta:
updated:
created:
...
version:
env: main # Is this a top level prop? I think so - all projects have an env now
openfn:
uuid:
domain:
sandbox:
parent-name # I want the file to relate to the parent project by a useful name
parent-uuid: # remember that this is a UUID to a Project, not an id
color:
...
options:
allow_support_access
concurrency
...
credentials:
...
collections:
...
workflows:
- id:
name:
version-history:
openfn:
- uuid:
meta:
updated:
created:
options:
concurrency
Some of these properties will not be available in the provisioner v1. That's fine: we map what we can and add support for the rest later.
I'm not sure about the version key. There are possibly a few versions:
- What version of local project spec are we using?
- What provisioner API version generated the state?
- What Lightning version was the project last downloaded from? Maybe save this this in
openfn
What I really want is to know what local Project Spec version was used, so that it can be correctly parsed by the CLI. I think this version key is also the thing that tells me it's a v2 project file.
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Activity
josephjclark commentedon Oct 22, 2025
Project.serialize('state')ALWAYS means "old v1 app state". So if I serialise to state, it must be 100% compatible as JSON or YAMLProject.serialise('yaml'). At this point we're using the new, proprietary v2 format on disk.serialize('json')should use the same structure, just in JSON.josephjclark commentedon Nov 18, 2025
pontificating of the complexities of the implementation and I think I've got it;
version: 2keyNow
from('project')works on all project/state files.from('yam')is just a short hand forfrom('project', { format: 'yaml' }from('file')will always load the contents and callfrom('project')and we know we'll be compatiblejosephjclark commentedon Nov 18, 2025
Wondering aloud where the version key serializes:
.version- just drop it on the top.meta.version- probably here, in this box, with maybelast_sync_datekind of stuff. Meta is for stuff only the CLI cares about.config.versionbut it's not a workspace config thing - project files might be different versionsjosephjclark commentedon Nov 21, 2025
workflows are a bit of a problem
The obvious thing to do is serialize with teh v1 state structure
{ id: '<uuid>', name, trigger, jobs, steps }. This is a lightning-facing file after all.Workflow.yaml should probably basically use the same structure for consistency (even if the structures are harder to edit by hand).I want users to be able to copy and paste workflows to/from the app.
The problem is that the
idused by lightning state is a UUID. And to keep things clear and crisp, I don't want any UUIDs in any CLI stuff to be called ids. I want to differentiate.Ah but the app's workflow.yaml is the answer: it uses human ids and excludes uuids. I should serialize to this structure, putting all options and uuids onto the
.openfnobject