diff --git a/.github/community/01-crd-aligned-resource-schema.md b/.github/community/01-crd-aligned-resource-schema.md new file mode 100644 index 0000000..c6d9cd6 --- /dev/null +++ b/.github/community/01-crd-aligned-resource-schema.md @@ -0,0 +1,425 @@ +# 01 — CRD-Aligned Resource Schema + +> **Goal.** Reshape the firestone resource document so it reads like a +> Kubernetes CRD: a typed envelope (`apiVersion` / `kind` / `metadata` / +> `spec`), all human-facing prose consolidated under `metadata`, and +> identity/naming handled by a `names` block instead of overloading `kind`. +> +> **Stop condition.** `firestone generate …` accepts the new document shape +> for every generator (openapi, asyncapi, cli-python, cli-rust, streamlit); +> `firestone convert` rewrites a legacy file into the new shape; the examples +> and docs use the new shape; legacy files still generate byte-identical +> output behind a deprecation warning. + +> **Status:** ⛔ Not started. Analysis below verified against `main` @ +> `c301292` on 2026-09-16. + +--- + +## 1. Why this exists + +The resource document grew one key at a time. It works, but the top level is +now a flat bag of fifteen sibling keys that mix five unrelated concerns: + +| Concern | Keys today | +|---|---| +| Identity | `kind`, `apiVersion`, `singular`, `plural` | +| Routing | `versionInPath` | +| Human prose | `metadata`, `descriptions`, and `descriptions` again at two deeper levels | +| API policy | `methods`, `security`, `default_query_params` | +| Code generation | `client_pkg`, `namespace`, `model_pkg_overrides` | +| The actual schema | `schema` | + +Nothing tells a reader which of those are about the API being described and +which are about firestone's own output. A CRD makes that split structural: +the envelope identifies the *document*, `metadata` describes it to humans, and +`spec` holds everything about the thing being defined. + +### 1.1 Concrete problems + +**`metadata` is dead weight.** It is declared in +[`firestone/schema/resource.yaml:12`](../../firestone/schema/resource.yaml) +with `name` and `description`, it appears in all four example resources — +and **no generator reads it**. Confirmed by grep across `firestone/spec/` and +`firestone/schema/*.jinja2`: zero hits. + +Worse, [`docs/…/resource-schema/metadata.md`](../../docs/site/content/core-concepts/resource-schema/metadata.md) +documents a pipeline that does not exist — it claims `metadata.description` +becomes OpenAPI `info.description`. In reality `info.description` comes from +the **required** CLI flag `--description` +([`firestone/__main__.py:35`](../../firestone/__main__.py) → +[`openapi.jinja2:4`](../../firestone/schema/openapi.jinja2)). The document +that should own the prose cannot supply it. `README.md:243` repeats the claim +("rendered in generated documentation, UI titles, and CLI help"). + +**Prose is scattered across four levels.** Operation descriptions live in +top-level `descriptions.{resource,instance}.{get,…}`, then again in +`schema.descriptions`, then again in `schema.items.descriptions`, alongside +per-property `description`. Generators spend real code deleting them back out +before emitting components: + +- `firestone/spec/openapi.py:156`, `:417-420`, `:538` +- `firestone/spec/asyncapi.py:446` + +**`kind` is overloaded four ways.** It is the plural URL segment, the OpenAPI +tag, the generated module name, and the input to +`spec_base.to_singular()` ([`_base.py:33`](../../firestone/spec/_base.py)) for +component names. `singular:` (#85) and `plural:` (#89) were added as escape +hatches when the derivation guessed wrong. A CRD says this directly: + +```yaml +names: + kind: Address + listKind: AddressList + singular: address + plural: addresses +``` + +**`apiVersion` is not an API version.** `apiVersion: v1` is only consulted +when `versionInPath: true`, to build a `/v1.0/` path prefix +(`openapi.py:602`). There is no group, no served/storage notion, no way to +describe two versions of one resource. Meanwhile `namespace:` (#88) was +invented to prefix generated module names so two projects can share a +`kind` — which is exactly what a CRD **group** is for. + +**Field naming is inconsistent.** `apiVersion` and `versionInPath` are +camelCase; `default_query_params`, `query_params`, `instance_attrs`, +`client_pkg`, `model_pkg_overrides` are snake_case. Kubernetes is camelCase +throughout. + +**Property shape is inconsistent.** Some properties are inline JSON Schema +(`addrtype: {type: string}`), others wrap in `schema:` (`person: {schema: +{$ref: …}}`), and firestone adds a bare `expose: false` keyword that is not +JSON Schema. Kubernetes keeps the schema pure OpenAPI v3 and puts extensions +behind an `x-kubernetes-*` prefix. + +### 1.2 What we are *not* doing + +- Not becoming a controller, and not adopting reconciliation semantics. +- Not adopting `status` subresources in this roadmap (see §8, deferred). +- Not implementing real multi-version serving — the `versions` list lands as + a one-element list (§3.3) so the shape is right; serving two versions at + once is a separate roadmap. +- Not changing generated output. Every phase here is shape-only: the + OpenAPI/AsyncAPI/CLI bytes produced from a converted file must match what + the legacy file produced. + +--- + +## 2. Target document shape + +```yaml +apiVersion: firestone.dev/v1alpha1 # version of the *document format* +kind: Resource # this document defines one resource + +metadata: + name: addressbook # document name (was: kind) + title: Address Book # feeds info.title / tag name + summary: Addresses and the people at them + description: | # feeds info.description / tag description + An example of an addressbook resource. + descriptions: # ALL operation prose, one place + resource: + get: List all addresses in this addressbook. + post: Create a new address in this addressbook. + instance: + get: Get a specific address from this addressbook. + delete: Delete an address from this addressbook. + instanceAttrs: {} + labels: {} # free-form, passed through to x-firestone-labels + annotations: {} + +spec: + group: addressbook.example.com # replaces `namespace:`; optional + names: + kind: Address # singular PascalCase -> component names + plural: addresses # URL segment, module name, tag + singular: address + versions: + - name: v1 + served: true + inPath: false # was: versionInPath + schema: + key: + name: address_key + description: A unique identifier for an addressbook entry. + schema: {type: string} + queryParams: [] # was: schema.query_params + openAPIV3Schema: # was: schema.items + type: object + properties: {} + required: [] + methods: + resource: [get, post] + instance: [delete, get, head, put] + instanceAttrs: [delete, get, head, put] + queryParams: + default: [] # was: default_query_params + security: + schemes: {} # was: security.scheme + resource: [post] + instance: [delete, put] + asyncapi: + servers: {} + channels: {} + codegen: # firestone's own output knobs, isolated + rust: + clientPkg: addressbook_client # was: client_pkg + modelPkgOverrides: {} # was: model_pkg_overrides + modulePrefix: project_a # was: namespace (if group is not used) +``` + +Three things this buys immediately: + +1. **One place for prose.** `metadata` owns everything a human reads about + the resource. Property-level `description` stays inline in the schema, + because that is where OpenAPI puts it. +2. **Identity stops being guesswork.** `names.kind` / `names.singular` / + `names.plural` are explicit; `to_singular()` becomes a *default* used only + to fill in missing `names` entries, not a load-bearing heuristic. +3. **Codegen knobs stop polluting the API description.** `spec.codegen` is + the only place a Rust crate name may appear. + +### 2.1 Full field mapping + +| Legacy | New | Notes | +|---|---|---| +| *(none)* | `apiVersion` | `firestone.dev/v1alpha1` — required, identifies the format | +| *(none)* | `kind` | Always `Resource` in this roadmap | +| `kind: addressbook` | `metadata.name` + `spec.names.plural` | Same string by default | +| `singular` | `spec.names.singular` | | +| `plural` | `spec.names.plural` | | +| *(derived)* | `spec.names.kind` | PascalCase singular; defaults from `to_singular()` | +| `apiVersion: v1` | `spec.versions[0].name` | | +| `versionInPath` | `spec.versions[0].inPath` | | +| `namespace` | `spec.group` (preferred) or `spec.codegen.rust.modulePrefix` | | +| `metadata.description` | `metadata.description` | Unchanged key — now actually read | +| `metadata.name` | `metadata.name` | Now actually read | +| *(CLI `--title`)* | `metadata.title` | Flag becomes an optional override | +| *(CLI `--summary`)* | `metadata.summary` | | +| `descriptions.*` | `metadata.descriptions.*` | `instance_attrs` → `instanceAttrs` | +| `schema.descriptions` | `metadata.descriptions.resource` | Merged; legacy wins on conflict during conversion, with a warning | +| `schema.items.descriptions` | `metadata.descriptions.instance` | Same | +| `methods` | `spec.methods` | `instance_attrs` → `instanceAttrs` | +| `security.scheme` | `spec.security.schemes` | | +| `security.{resource,…}` | `spec.security.{resource,…}` | | +| `default_query_params` | `spec.queryParams.default` | | +| `schema.key` | `spec.versions[0].schema.key` | | +| `schema.query_params` | `spec.versions[0].schema.queryParams` | | +| `schema.items` | `spec.versions[0].schema.openAPIV3Schema` | | +| `schema.type: array` | *(dropped)* | Implicit: a resource is a collection | +| `asyncapi` | `spec.asyncapi` | | +| `client_pkg` | `spec.codegen.rust.clientPkg` | | +| `model_pkg_overrides` | `spec.codegen.rust.modelPkgOverrides` | | +| `expose: false` | `x-firestone-expose: false` | Legacy key accepted indefinitely | + +--- + +## 3. Design decisions to lock first + +These belong in an ADR before any code moves. + +### 3.1 Group / domain + +`firestone.dev` is a placeholder. Options: `firestone.jeb.ca`, `firestone.io`, +or a domain that is actually controlled. Pick one — it goes in every resource +file forever. + +### 3.2 `kind: Resource` vs `kind: ` + +Two readings of "CRD-like": + +- **(a) Envelope, recommended.** The document's `kind` is `Resource`; the API + resource's kind lives in `spec.names.kind`. Matches how a + `CustomResourceDefinition` works, and makes `apiVersion`/`kind` describe the + *file*, which is what lets firestone version its own format. +- **(b) Instance-style.** Keep `kind: addressbook` at the top and add + `spec`/`metadata` around it. Less churn, but then `apiVersion` is ambiguous + (is it the format version or the API version?) — the exact confusion this + roadmap is trying to remove. + +Go with (a). + +### 3.3 `versions` as a list + +A list is CRD-shaped and future-proof, but every generator currently assumes +one version. Land it as a **list with exactly one served entry**, validated by +the schema (`maxItems: 1` under v1alpha1). Generators read the served entry. +Relaxing `maxItems` later is a schema change with no generator rewrite. + +### 3.4 Compatibility window + +Recommended: **dual-read with a deprecation warning.** Legacy documents keep +working for at least two minor releases, `firestone convert` does the rewrite, +and a `FIRESTONE_STRICT=1` / `--strict` flag turns the warning into an error +for projects that want to enforce the new shape in CI. + +### 3.5 Where validation lives + +Today `firestone_lib.resource.validate()` hardcodes +`importlib.resources.files("firestone.schema")/"resource.yaml"` — the library +reaches back into *this* package for the schema. With two schemas to choose +between, dispatch belongs here, not there. + +Recommendation: add `firestone/resource/validate.py` that picks the schema by +document shape (`apiVersion` starting with the firestone group → v1alpha1, +otherwise legacy) and call it from `__main__.py`. Leave +`firestone_lib.resource.validate()` alone so existing firestone-lib consumers +are unaffected; `get_resource_schema()` (the `$ref` resolver) is still used +as-is. + +--- + +## 4. Phases + +### Phase 1 — Internal resource model (enabling refactor) + +**Nothing user-visible. Do this first.** Generators read raw dicts in 59 +places (`asyncapi` 5, `cli` 11, `cli_rust` 20, `openapi` 13, `streamlit` 10). +Changing the document shape while those reads are spread across five modules +means touching all of them twice. + +- [ ] Add `firestone/resource/model.py` with a frozen dataclass + (`Resource`, `ResourceNames`, `ResourceVersion`, `Descriptions`, + `Codegen`) — no Pydantic dependency needed, but `pydantic` is already a + runtime dep if a validating model is preferred. +- [ ] Add `Resource.from_legacy(dict)` implementing the §2.1 mapping. +- [ ] Replace every `rsrc["kind"]` / `rsrc.get("plural")` / … in + `firestone/spec/*.py` with attribute access on the model. +- [ ] Keep `to_singular()` as the fallback used only when `names.singular` is + absent. +- [ ] Golden test: for each example resource, generated openapi/asyncapi/cli + output is byte-identical before and after the refactor. + +*Effort: the bulk of this roadmap. ~2,900 lines of generator code, ~2,000 +lines of tests referencing resource dicts.* + +### Phase 2 — The v1alpha1 schema + +- [ ] `firestone/schema/resource-v1alpha1.yaml` — the new meta-schema, + `additionalProperties: false` at every level, camelCase throughout. +- [ ] `firestone/resource/validate.py` — shape dispatch (§3.5), clear error + when a legacy key appears in a v1alpha1 document and vice versa. +- [ ] `Resource.from_v1alpha1(dict)` next to `from_legacy`. +- [ ] Deprecation warning on the legacy path, naming the file and pointing at + `firestone convert`. +- [ ] Tests: both shapes of the same logical resource produce an identical + `Resource`. + +### Phase 3 — Make `metadata` real + +This is the part that closes the documented-but-missing behaviour. + +- [ ] `info.title` / `info.description` / `info.summary` fall back to + `metadata.title` / `.description` / `.summary`; CLI flags become + optional overrides (`--title`, `--description` lose `required=True`). +- [ ] Multi-resource generation: each resource's `metadata.description` + becomes its OpenAPI **tag** description (currently tags carry no + description at all). +- [ ] `metadata.descriptions` replaces the three scattered description + blocks; the `del … ["descriptions"]` cleanup code in `openapi.py` and + `asyncapi.py` disappears with them. +- [ ] Same wiring for asyncapi `info`, the Rust/Python CLI help text, and the + Streamlit page headings. +- [ ] Error, don't silently ignore, when neither a flag nor `metadata` + supplies a title/description. + +### Phase 4 — `firestone convert` + +- [ ] `firestone convert -r legacy.yaml [-O new.yaml] [--in-place]`, + round-trip safe, comment loss documented (PyYAML drops comments — + consider `ruamel.yaml` as a dev-only dependency to keep them). +- [ ] Emits warnings for the lossy cases: merged description blocks, + `namespace` → `group` vs `modulePrefix`, `expose` → `x-firestone-expose`. +- [ ] Convert `examples/addressbook/*.yaml` and + `examples/addressbook_rs/contacts.yaml`; regenerate the checked-in + `openapi.yaml` / `asyncapi.yaml` and diff them to prove no drift. +- [ ] Keep one legacy example under `examples/legacy/` so the compat path + stays tested. + +### Phase 5 — Documentation + +- [ ] Rewrite `docs/site/content/core-concepts/resource-schema/` — it is + organised one-page-per-top-level-key (`kind.md`, `apiversion.md`, + `metadata.md`, `version-in-path.md`, …), which maps onto the *old* + shape. New structure: `envelope.md`, `metadata.md`, `spec-names.md`, + `spec-versions.md`, `spec-methods.md`, `spec-security.md`, + `spec-codegen.md`. +- [ ] New page: `core-concepts/resource-schema/migrating-to-v1alpha1.md` + carrying the §2.1 table. +- [ ] Fix `metadata.md`'s claim about `info.description` — true only after + Phase 3. +- [ ] Update `README.md` — `README.md:243` makes the same false claim as the + docs ("rendered in generated documentation, UI titles, and CLI help") — + and `changelog.md`. +- [ ] Refresh every YAML block in `getting-started/` and `examples/`. + +### Phase 6 — Deprecation + +- [ ] Legacy warning becomes loud (once per file, with the converted path + suggested). +- [ ] `--strict` / `FIRESTONE_STRICT=1` promotes it to an error. +- [ ] Announce removal target in the changelog; remove `from_legacy` and + `resource.yaml` no earlier than two minor releases later. + +--- + +## 5. Risks + +| Risk | Mitigation | +|---|---| +| Silent output drift during the Phase 1 refactor | Golden-file tests captured **before** any change; byte-comparison in CI | +| firestone-lib coupling (`validate()` reaches into `firestone.schema`) | Keep `resource.yaml` in place and unchanged; dispatch in firestone (§3.5) | +| Downstream projects with many legacy resource files | `firestone convert`, plus a long dual-read window | +| `$ref` across resource files (`person.yaml#/schema`) breaks | The `$ref` target path changes to `#/spec/versions/0/schema/openAPIV3Schema`; `convert` must rewrite intra-repo `$ref`s, and this needs its own test — `jsonref` resolves these at load time in `firestone_lib.get_resource_schema()` | +| Comment loss on conversion | `ruamel.yaml` round-trip, or document it and keep conversion opt-in | +| Bikeshedding the group domain | Decide in the ADR (§3.1) before Phase 2 | + +The `$ref` row is the sharpest edge: `examples/addressbook/addressbook.yaml` +embeds `$ref: "person.yaml#/schema"`, and that pointer is spelled against the +*legacy* layout. Every downstream schema doing the same breaks on conversion +unless `convert` rewrites it. + +--- + +## 6. Definition of done + +1. All five generators run off `firestone.resource.model.Resource`; no + generator reads a raw resource dict. +2. `resource-v1alpha1.yaml` validates the new shape; legacy files still + validate and still generate identical output, with a deprecation warning. +3. `metadata` drives `info`, tag descriptions and operation prose; `--title` + and `--description` are optional. +4. `firestone convert` converts all four examples, including `$ref` rewrites, + with regenerated specs diffing clean. +5. `core-concepts/resource-schema/` documents the new shape and carries a + migration page. +6. An ADR records §3.1–§3.5. + +--- + +## 7. Open questions + +1. **Group domain** — what goes in `apiVersion` (§3.1)? +2. **Does `spec.group` replace `namespace` entirely**, or do both survive + (group for API identity, `modulePrefix` for Rust module names)? +3. **`listKind`** — worth carrying for a REST tool, or Kubernetes cargo cult? +4. **Should `scope` exist?** CRDs have `Namespaced`/`Cluster`. The REST + analogue would be "is this resource nested under a parent path", which + firestone currently expresses through `instance_attrs` recursion. +5. **Is `x-firestone-*` the right extension prefix**, given generated OpenAPI + already passes unknown keys through? + +--- + +## 8. Deferred (possible follow-on roadmaps) + +- **Multi-version serving** — relax `versions` to N entries, generate + `/v1/…` and `/v2/…` path trees from one document. +- **`status` sub-schema** — a CRD-style `status` block generating a + read-only projection plus `PATCH /status` semantics. +- **Printer columns** — `additionalPrinterColumns` is a natural fit for the + Streamlit table and the CLI's output columns, which today are supplied + out-of-band via `--col-mappings`. +- **Conditions** — a standard `conditions[]` shape for long-running + operations. diff --git a/.github/community/02-rust-rewrite.md b/.github/community/02-rust-rewrite.md new file mode 100644 index 0000000..c1d9a98 --- /dev/null +++ b/.github/community/02-rust-rewrite.md @@ -0,0 +1,431 @@ +# 02 — Port the firestone CLI to Rust + +> **Goal.** Move all of `./firestone` — every `generate` subcommand, from +> resource loading through OpenAPI, AsyncAPI, the CLI generators, streamlit, +> validations and the rust-axum server glue — from Python to Rust, keeping the +> Jinja2 template set and the generated artifacts intact. +> +> **Stop condition.** Every `firestone generate …` subcommand is served by +> Rust; the differential harness reports no semantic difference against the +> Python implementation across the resource corpus; `firestone/`, `test/`, +> `pyproject.toml` and `poetry.lock` are deleted and the release pipeline +> ships binaries. + +> **Status:** ⛔ Not started. Analysis verified against `main` @ `c301292` +> plus the uncommitted `validations` / `server_rust` work in the tree on +> 2026-09-16. + +--- + +## 1. Scope + +The whole CLI. Seven subcommands across eleven modules: + +| Module | Lines | Subcommand | +|---|---|---| +| [`spec/openapi.py`](../../firestone/spec/openapi.py) | 725 | `generate openapi` | +| [`spec/asyncapi.py`](../../firestone/spec/asyncapi.py) | 477 | `generate asyncapi` | +| [`spec/cli.py`](../../firestone/spec/cli.py) | 378 | `generate cli -l python` | +| [`spec/cli_rust.py`](../../firestone/spec/cli_rust.py) | 912 | `generate cli -l rust` | +| [`spec/streamlit.py`](../../firestone/spec/streamlit.py) | 364 | `generate streamlit` | +| [`spec/validations.py`](../../firestone/spec/validations.py) | 654 | `generate validations` + the `x-firestone-validations` the OpenAPI document carries | +| [`spec/server_rust.py`](../../firestone/spec/server_rust.py) | 504 | `generate server` | +| [`spec/_base.py`](../../firestone/spec/_base.py) | 61 | Jinja env, `yaml_pretty`, `pascal`, `to_singular`, `get_opid` | +| [`__main__.py`](../../firestone/__main__.py) | 495 | The click surface, resource loading, validation | +| `spec/__init__.py` | 21 | — | +| `firestone-lib` | 6 symbols | `init_logging`, `PathList`, `AnyDict`, `get_resource_schema`, `validate`, `utils` | + +**4,591 lines of Python**, plus 3,902 lines of tests that get rewritten but +are worth mining for corpus cases first. + +### 1.1 What is kept + +- **`firestone/schema/**/*.jinja2` — 4,126 lines across 30 templates.** More + than half the lines in the project and all of the output fidelity. MiniJinja + is close enough to Jinja2 that most should compile unchanged; §4.5 is about + proving that in week one rather than assuming it. +- **`firestone/schema/resource.yaml` (293 lines)** — the meta-schema is data, + loaded from the crate as-is. + +### 1.2 What does not change + +- **No output changes.** This is a port, not a redesign. Roadmap + [01](01-crd-aligned-resource-schema.md) is where the document shape moves; + nothing here alters what firestone emits, beyond the one-time YAML + formatting re-baseline in §4.1. +- **No new generators**, and no generator dropped without a decision (§9). +- **No permanent hybrid.** A Python/Rust CLI held together by PyO3 is worse + than either language alone. Both implementations coexist only while the + phases run, with the Python one as the oracle. + +--- + +## 2. Why + +**Most of what firestone emits is already Rust.** `cli_rust.py` (912), +`server_rust.py` (504), the `validations/rust/` templates, +`Cargo.toml.jinja2` and the whole `examples/addressbook_rs/` tree exist to +generate Rust. The tool that generates Rust for Rust projects is a Python +package those projects cannot depend on, build, or vendor. + +**Distribution is the real pain.** `pyproject.toml` carries five dependency +groups (`asyncapi`, `dev`, `build`, `fastapi`, `webui`) and two extras +(`caching`, `validations`) over a `>=3.12,<3.14` interpreter range, plus a +separately released `firestone-lib` that reaches back into `firestone.schema` +for its meta-schema. Generating a Rust server currently requires a Python +toolchain with the right extras selected. One binary removes that whole class +of problem. + +**A typed model comes free.** The generators are dict-shuffling code: +`copy.deepcopy(schema["items"])`, in-place `del schema["descriptions"]`, +`rsrc.get("plural") or rsrc["kind"]`. Roadmap 01 exists largely to introduce a +typed internal model; in Rust that model is the deserializer, not a layer over +one (§6). + +**Speed is not the argument.** Generation is sub-second today. Do not put +performance in the pitch. + +--- + +## 3. Crate layout + +``` +crates/ + firestone-core/ loading, $ref resolution, meta-schema validation, + the resource model (firestone-lib + _base.py) + firestone-openapi/ the OpenAPI emitter (openapi.py) + firestone-asyncapi/ the AsyncAPI emitter (asyncapi.py) + firestone-validations/ rule normalisation + package emission (validations.py) + firestone-codegen/ cli-python, cli-rust, streamlit, server-rust + firestone-templates/ the .jinja2 set, embedded with include_dir! + firestone-cli/ the `firestone` binary (clap) (__main__.py) +xtask/ differential harness + corpus runner +``` + +`firestone-templates` is separate on purpose: it keeps templates loadable both +from the embedded set and from a directory, which is what `--template` already +allows. + +| Python | Rust | Notes | +|---|---|---| +| `click` | `clap` (derive) | `PathList` / `AnyDict` become value parsers | +| `Jinja2` | `minijinja` | Needs `loop_controls` — `_base.py` enables `jinja2.ext.loopcontrols` | +| `pyyaml` | `serde_yaml_ng` / `saphyr` | `serde_yaml` is unmaintained; emitter fidelity is §4.1 | +| `jsonschema` | `jsonschema` crate | Draft and error-text differences (§4.4) | +| `jsonref` | hand-rolled resolver | §4.2 | +| `pydantic` | `serde` | Only used *by generated* Python validation packages | +| `quart` + `swagger-ui-py` | `axum` + `utoipa-swagger-ui` | Only for `--ui-server` | +| `cel-python` | `cel-interpreter` | Test-side only (§4.6) | +| `firestone-lib` | folded into `firestone-core` | §9 | + +--- + +## 4. The hard parts + +Everything else is mechanical. These are not. + +### 4.1 PyYAML's emitter is part of the contract + +The checked-in `examples/addressbook/openapi.yaml` is PyYAML output and +carries PyYAML's choices: + +```yaml + CreateAddressbook: + allOf: + - $ref: '#/components/schemas/addressbook' # list item NOT indented under its key +``` + +- **Keys are sorted.** `yaml.dump(data, indent=2)` in + [`_base.py:23`](../../firestone/spec/_base.py) defaults to `sort_keys=True` + — that is why `CreateAddressbook` / `CreatePerson` / `CreatePostal_code` + come out alphabetically rather than in resource order. +- **Sequences are not indented** relative to their parent key. Rust YAML + emitters indent them. +- **Scalars starting with `#` get single-quoted**; others do not. +- **Lines wrap at 80 columns** by default, mid-description. + +Byte-identical YAML means reimplementing PyYAML's emitter. Don't. + +**Decision: YAML and JSON outputs are compared semantically** (parse both, +compare values), with the checked-in example specs re-baselined once in a +single commit whose diff is pure formatting. **Template-rendered output — +every `.py`, `.rs`, `Cargo.toml` — is compared byte-for-byte**, since MiniJinja +renders the same text the same way. + +That re-baseline is the only user-visible change in this roadmap and belongs +in the changelog: anyone diffing firestone output in CI sees it once. + +### 4.2 `$ref` resolution, and un-resolution + +`firestone_lib.resource.get_resource_schema()` resolves `$ref` at load time +with `jsonref`, including cross-file refs like `$ref: "person.yaml#/schema"` +against a `file:` base URI. + +The subtlety: **`cli_rust.py` needs the `$ref` back after jsonref destroyed +it.** `cli_rust.py:429-438` stashes `original_schema` before resolution so the +Rust generator can tell "this property is a `Person`" from "this property is +an inline object" — `model_pkg_overrides` keys off the `$ref` filename stem. + +The Rust resolver must produce a structure carrying *both* the resolved value +and the originating ref, rather than mimicking jsonref's lossy proxies. A +design improvement disguised as a port obligation. + +### 4.3 Ordering + +Python dicts preserve insertion order and the generators rely on it for +everything that does not pass through `yaml.dump` — paths, parameter lists, +the `x-firestone-validations` block, generated match arms. Rust needs +`serde_json`'s `preserve_order` feature or `IndexMap` throughout, or output +order becomes hash order. Easy to get wrong, invisible until a diff appears. + +### 4.4 Schema validation error text + +`firestone_rsrc.validate()` surfaces `jsonschema` messages straight to the +user, and `test_validations.py` asserts on firestone's own +`InvalidValidation` text. Firestone's own errors are ours to reword; raw +schema-validation text will change, and the ported tests should stop asserting +on third-party message text. + +### 4.5 Template compatibility + +30 templates, 4,126 lines. Jinja2 features in use that need checking against +MiniJinja **before** committing to it: + +- the `loopcontrols` extension (`{% break %}` / `{% continue %}`) +- custom filters `yaml_pretty` and `pascal` (`_base.py:23,33`) and the + built-in `tojson` (`openapi.jinja2:3-4`) +- `select_autoescape()` behaviour — it returns *false* for `.jinja2` today, so + the Rust side must have autoescape **off** or every generated quote becomes + `"` +- whitespace control (`{%-`, `-%}`) and undefined-variable semantics, where + Jinja2 renders empty and MiniJinja can be made strict + +**This is Phase 1 work, not Phase 7 work.** If MiniJinja cannot render these, +the plan changes shape, and that should be known in week one. + +### 4.6 CEL + +Firestone never evaluates CEL — it transports expressions into +`x-firestone-validations` and the generated packages. But +`test_validations.py` (1,283 lines) *does*, through `cel-python`, to prove +each rule's `examples` pass and fail as declared. Ported, that becomes +`cel-interpreter`, and the two engines will not agree on every edge — `has()` +on absent optional fields is exactly what the addressbook example was written +to pin down. Budget a conformance pass over every example in the corpus. + +--- + +## 5. Strategy: differential, not big-bang + +The Python implementation is the oracle. Nothing is done because it looks +right; it is done when its output matches Python's across the corpus. + +``` +xtask diff --corpus corpus/ --python "poetry run firestone" --rust "target/debug/firestone" +``` + +For every resource document × every generator × every relevant flag +combination: run both, compare (semantically for YAML/JSON, byte-wise for +rendered code), report the first differing path. + +The corpus starts as `examples/addressbook/*.yaml`, +`examples/addressbook_rs/contacts.yaml` and the fixtures embedded in +`test/spec/*.py` — `test_openapi.py` (1,039), `test_cli_rust.py` (1,016) and +`test_validations.py` (1,283) are the real specification of this tool and +should be mined for cases, not merely ported. Every difference found becomes a +corpus entry before it is fixed. + +**A module is frozen once its phase starts.** A Python-side change during its +port is a change to the oracle; if one is unavoidable, it lands in the corpus +first. + +--- + +## 6. Phases + +Dependency-ordered. Each ends with a clean corpus diff for its surface, so a +stall leaves a working Python tool rather than a broken hybrid. + +### Phase 1 — Harness and template spike + +- [ ] `xtask diff` runner and the initial corpus, Python pinned as the oracle. +- [ ] Python's output for the whole corpus captured as committed goldens. +- [ ] **MiniJinja spike:** render all 30 templates against a fixed context and + diff (§4.5). Decide MiniJinja vs Tera vs Askama on evidence. +- [ ] Settle the YAML comparison policy (§4.1); land the one-time re-baseline + of `examples/**/openapi.yaml` and `asyncapi.yaml`. + +### Phase 2 — `firestone-core` + +- [ ] Load YAML/JSON; resolve `$ref` preserving the original ref (§4.2); + validate against `resource.yaml`. +- [ ] The typed resource model, named for roadmap 01's target vocabulary + (`names`, `versions`, `metadata`) while deserializing today's flat + shape, so 01 lands as a second `Deserialize` impl (§7). +- [ ] Absorb the six `firestone-lib` symbols, including the `utils` use at + `streamlit.py:13`. +- [ ] Port `_base.py`: `to_singular`, `pascal`, `get_opid`, `yaml_pretty`. +- [ ] `firestone-cli` skeleton: the full clap surface, every subcommand + erroring `not yet ported`. + +### Phase 3 — Rule normalisation + +Split from the rest of `validations.py` because the OpenAPI document depends +on it: `openapi.py` calls `extract`, `strip`, `methods_with_rules`, +`problem_response`, `problem_component` and `openapi_extension` +(`openapi.py:611,621,643,647,652,724`). + +- [ ] `references` desugaring into `exists` / `immutable` rules, `expr` rules, + method defaults, the `InvalidValidation` error set. +- [ ] Those six entry points, compared as data before any document renders. + +### Phase 4 — openapi + +The largest emitter (725 lines) and the one others read from — `server_rust.py` +consumes the document it produces. + +- [ ] Components: per-resource schemas, `Create*` / `Update*` variants, `$ref` + rewriting for nested resources. +- [ ] Paths: resource, instance, and the `instance_attrs` recursion. +- [ ] Parameters: keys, `default_query_params`, per-method `query_params`. +- [ ] Security schemes and per-method application. +- [ ] `x-firestone-validations` and the `ValidationProblem` component. +- [ ] `--prefix`, `--version` across 3.0.x / 3.1.x. Corpus diff clean. + +### Phase 5 — asyncapi + +- [ ] Channels for resource / instance / instance_attrs, servers block. +- [ ] Corpus diff clean. + +### Phase 6 — validations packages + +- [ ] Python and Rust package emission from the existing templates + (`validations/python/**`, `validations/rust/**`), `--no-tests`. +- [ ] CEL conformance pass over every declared example (§4.6). + +### Phase 7 — CLI generators + +- [ ] `cli.py` — Python/Click output, 378 lines. +- [ ] `cli_rust.py` — 912 lines; the `original_schema` / + `model_pkg_overrides` path from §4.2 is the risky part. +- [ ] `generate_cargo_toml`, multi-crate `client_pkg`, `--as-modules`. + +### Phase 8 — server and streamlit + +- [ ] `server_rust.py` (504), including its read-back of the generated + OpenAPI document. +- [ ] `streamlit.py` (364) and `--col-mappings`. +- [ ] `--ui-server` on axum, or drop it (§9). + +### Phase 9 — Cutover + +- [ ] Rewrite the four workflows: `build.yml`, `pr.yml`, `publish.yml` and + `docs.yaml` are poetry/pytest/pylint/black pipelines today. They become + `cargo test` / `clippy` / `fmt`, plus the corpus diff as a required + check. +- [ ] `docs.yaml` runs `pydoc-markdown` to build the API reference from Python + docstrings — that becomes `cargo doc`, or the API-reference section goes + away, since it documents internals rather than the CLI contract. +- [ ] Release path: platform binaries (`cargo-dist`) and the PyPI decision + (§8). +- [ ] Delete `firestone/`, `test/`, `pyproject.toml`, `poetry.lock` in one + commit, after one release where both exist. +- [ ] Rewrite installation docs — the Zola site's getting-started assumes pip. + +--- + +## 7. Sequencing against roadmap 01 + +These two overlap on exactly one thing: the typed model. Roadmap 01's Phase 1 +exists to introduce one in Python; this port gets it from +`#[derive(Deserialize)]`. + +**Recommendation: port first, land 01 in Rust.** Doing 01 in Python means +writing `from_legacy()`, the dual-read dispatch and the converter twice. After +this roadmap, 01 is two `Deserialize` impls and a `convert` subcommand over +types that already exist. + +The cost is a feature freeze on the Python tree for the duration — or +double-maintenance, which is worse. If a freeze is unacceptable, the +alternative is to land 01 fully in Python first and start the port from the +v1alpha1 shape, accepting the duplicated conversion work. What must **not** +happen is both at once. + +Either way, 01's open questions (group domain, `scope`, `listKind`) stay open: +they are document-design questions and the implementation language does not +touch them. + +--- + +## 8. Distribution + +Today: PyPI package `firestoned`, console script `firestone`, plus the +separately published `firestone-lib`. + +| Option | For | Against | +|---|---|---| +| **(a) maturin/PyO3 wheel** — keep `pip install firestoned` working, shipping a compiled binary | No break for existing users | Wheel-per-platform matrix; PyO3 glue for a CLI with no Python API | +| **(b) cargo + release binaries only** | Simplest to build and reason about | Breaks every existing `pip install` and any CI doing it | +| **(c) both, then drop the wheel** | Migration window | Two pipelines during the window | + +Recommend **(c)**: publish a wheel that is just the binary for one release +cycle, announce the deprecation in the changelog, then move to (b). + +--- + +## 9. Open questions + +1. **`firestone-lib`** — does anything outside this project depend on it? It + is separately released and its `validate()` reaches back into + `firestone.schema` for *this* project's meta-schema. Folding it in removes + that inversion, but only if nothing else consumes it. +2. **`--ui-server`** — port onto axum, or drop it? It is a developer + convenience any static Swagger UI covers. +3. **Streamlit generator** — it emits a Python web UI. Still earning its + place, or deprecated rather than ported? +4. **Python CLI generation** (`cli.py`) — still used, or superseded in + practice by the Rust generator? +5. **Binary naming during the window** — does the Rust binary take the + `firestone` name with the Python one renamed, or ship under a different + name until Phase 9? +6. **Minimum Rust version / edition**, and whether firestone and the crates it + generates must agree on one. + +--- + +## 10. Risks + +| Risk | Mitigation | +|---|---| +| Silent output drift | The corpus diff is a required CI check from Phase 1, before any generator is ported | +| MiniJinja can't render the templates | Spiked in Phase 1, when changing course is still cheap | +| Port stalls half-done | Phases are dependency-ordered and each ends with a clean corpus diff; a stall leaves a working Python tool | +| Feature freeze violated | Any Python-side change during the port lands in the corpus first, so it cannot be forgotten | +| CEL engine disagreement | Conformance pass in Phase 6 over every declared example | +| Pinned `pip install firestoned` breaks | §8 option (c), with a deprecation release | +| `firestone-lib` consumers outside this project | Audit before absorbing (§9) | + +--- + +## 11. Definition of done + +1. Every `firestone generate …` subcommand runs with no Python in the path. +2. The corpus diff is clean across every resource document, generator and flag + combination, and runs as a required CI check. +3. `firestone/`, `test/`, `pyproject.toml` and `poetry.lock` are gone; + `firestone-lib`'s six symbols live in `firestone-core`. +4. The example specs are re-baselined in one formatting-only commit, noted in + the changelog. +5. The resource model is named for roadmap 01's target shape, so 01 lands as a + deserializer rather than a refactor. +6. Releases ship platform binaries, with the PyPI path settled per §8. + +--- + +## 12. Found while surveying + +Not part of this roadmap, but do not port them forward as-is: five error +messages are plain strings that were meant to be f-strings, so they print the +literal `{yaml.dump(schema)}` — `openapi.py:474`, `asyncapi.py:395`, +`cli.py:252`, `cli_rust.py:646`, `streamlit.py:242`. Two characters per site, +worth fixing in Python now so the corpus locks in the corrected text. diff --git a/.github/community/README.md b/.github/community/README.md new file mode 100644 index 0000000..fe3d84a --- /dev/null +++ b/.github/community/README.md @@ -0,0 +1,40 @@ +# firestone Roadmap Index + +This directory holds firestone's roadmap documents. Each one describes a body +of work — *what* and *why*, with a task list and a definition of done. +[`../../ROADMAPS.md`](../../ROADMAPS.md) is the status board that indexes them +and carries the current completion state. + +## Numbering + +Numbers are sequential and stable once assigned: the next roadmap is the next +free number, regardless of subject. firestone is small enough that grouping +numbers into themed bands (the way bindy does) would be ceremony — introduce +bands here only if this set grows past a dozen documents, and renumber nothing +when that happens. + +## Index + +| # | File | What | +|---|---|---| +| 01 | [`01-crd-aligned-resource-schema.md`](01-crd-aligned-resource-schema.md) | Reshape the resource document as a CRD-style envelope; make `metadata` the single home for prose | +| 02 | [`02-rust-rewrite.md`](02-rust-rewrite.md) | Port the whole CLI from Python to Rust, keeping the Jinja2 template set and the generated output | + +## Conventions + +- **Roadmaps say what and why.** An architecturally significant *how* — the + group domain, the compatibility window, where validation lives — is settled + in an ADR first; a roadmap entry does not substitute for one. +- **Task lists are the source of truth.** Check items off in the file as they + land, in the same PR that lands them. +- **Status changes go in `ROADMAPS.md`** in that same PR. That file is a + board, not documentation of intent. + +## Adding a roadmap + +1. Take the next free number. +2. Filename: `NN-lowercase-hyphenated-title.md`. +3. Open with a `> **Goal.**` / `> **Stop condition.**` block so a reader knows + what "done" means before reading the analysis, then a `> **Status:**` line + recording what was verified and against which commit. +4. Add a row to the table above **and** to `ROADMAPS.md`. diff --git a/ROADMAPS.md b/ROADMAPS.md new file mode 100644 index 0000000..ef39e7f --- /dev/null +++ b/ROADMAPS.md @@ -0,0 +1,35 @@ +# Roadmaps + +High-level index of firestone's roadmap documents. Full detail for each item +lives in [`.github/community/`](.github/community/) — this file tracks what +each one is and its current completion status; the detailed task lists and +design rationale live in the linked doc itself. + +Architecturally significant work in any roadmap below still goes +**ADR → tests → implement → docs**, in that order — a roadmap entry describes +*what* and *why*, it does not skip the ADR for *how*. + +## Status legend + +| Symbol | Meaning | +|---|---| +| ✅ | Done — implemented, tested, in the codebase today | +| 🔶 | In progress — some of it exists, not complete | +| ⛔ | Not started | +| 📄 | Reference doc — not a phase with a completion state | + +## Index + +Statuses were verified against `main` @ `c301292` on 2026-09-16. + +| # | Roadmap | Status | Notes | +|---|---|---|---| +| [01](.github/community/01-crd-aligned-resource-schema.md) | CRD-aligned resource schema | ⛔ | Top level is a flat bag of 15 keys; `metadata` is declared in `firestone/schema/resource.yaml:12` and read by **zero** generators, though the docs and `README.md:243` both claim it feeds `info.description`. Reshape into `apiVersion`/`kind`/`metadata`/`spec`, move all prose under `metadata`, replace the overloaded `kind` with a `names` block. Blocked on an ADR for the group domain and the compatibility window | +| [02](.github/community/02-rust-rewrite.md) | Port the CLI to Rust | ⛔ | All seven `generate` subcommands: 4,591 lines of Python across 11 modules, plus `firestone-lib`. The 4,126 lines of Jinja2 templates survive the port. Most of what firestone emits is already Rust, but the generator needs a Python toolchain and the right extras. Dependency-ordered phases — core, then openapi, then the rest — gated by a differential harness with the Python implementation as the oracle. Sequencing against 01 in its §7 | + +## Keeping this current + +When a roadmap item's status changes (something lands, something new starts), +update its row here in the same PR/commit that makes the change — this file is +a status board, not documentation of intent. Detailed task-level tracking +stays inside each roadmap doc; this file only tracks the item-level state.