Skip to content

Add the pre-flight check of a draft - #2556

Merged
pournasserian merged 3 commits into
v2from
2554-check-draft
Oct 19, 2026
Merged

pournasserian merged 3 commits into
v2from
2554-check-draft

Conversation

@pournasserian

Copy link
Copy Markdown
Member

Closes #2554

What

The pre-flight slice of F-CHK (#2420), CHK-04: an agent runs the site's active checks on a draft without storing it or opening a change set. It comes before the change set run (the epic's fifth slice), which waits on a decision about who the check engine acts as. The runner here is the one that run will use.

  • Core (FluentCMS.Core.Checks):
    • ICheckRunner runs every active check whose scope applies to a piece of content, at its active version and in name order. Each check sees the fields its scope names, and calls are charged to the caller. CheckSelection.Deterministic leaves out LLM-judged checks, and the run names the ones it left out (CheckRun.Skipped).
    • CheckResult holds the check, version, type, kind, severity, description, status, findings and model call.
    • CheckStatus is pass, warn (a failing check of warn severity), fail or needs review. CheckResult.Overall is fail if any check fails, then needs review, then warn, then pass.
  • Tool check_draft (FluentCMS.Core.Builtin.Tools.CheckTools):
    • Arguments: content_type, fields, optional sources, checks (all or deterministic), and id for a draft of an update.
    • It returns status, results (each with its findings and model call), skipped, and fieldErrors, the errors create_entry would give the draft.
    • With id, a check that asks about changes, such as sources, sees only the fields whose value differs from the entry's latest draft.
    • A draft with field errors fails. An unknown content type or entry is refused, with a hint for the call that finds one.
    • It stores nothing and opens no set.
  • MCP-01 coverage: "Run the checks on a draft before proposing it". CONTRIBUTING.md: the runner and the tool.

Why

CHK-04 asks for a pre-flight tool, so an agent fixes what the checks find before it proposes the change, instead of after a set fails. S2 is that loop: three findings, three fixes, a pass.

Choices worth a look:

  • Before the change set run. The run on every set (CHK-03) must move a set from checks running to checks failed or awaiting review. AgentPermissions refuses those moves for agents, and a job (Add the job queue with leases, retries and progress #2536) runs as whoever enqueued it, the submitting agent. Initiators today are only users and agents, so the run needs a decision on whom it acts as in the audit log. I have asked; this slice does not depend on the answer.
  • Not read-only. The tool stores nothing, but an LLM-judged check spends the agent's model budget, so it does not carry the read-only annotation that clients may approve without asking. checks: "deterministic" makes a free, instant call.
  • Field errors fail the draft. A draft the content type refuses cannot be proposed, so it never reads as a pass, whatever the checks say.
  • Updates. Without id, every field counts as changed, as for a new entry. With it, a headline edit on a priced page does not ask for a source for the unchanged price. The entry is read through IEntryDirectory, so the agent's scope applies.
  • Scope. The content type is read through IContentTypeDirectory, so a type outside the agent's scope is refused, and the result shows no check's settings or examples.
  • Not in this PR: the check run on every change set with results stored on the set, the tool that reads a set's results, and checks that read entities (F-ENT).

Tests

Unit:

  • CheckToolsConformanceTests (ToolProviderConformanceTests).
  • CheckRunner:
    • every active check that applies gives its status and evidence, in name order; a warn check's failure only warns
    • scope by content type and field, and an untyped draft
    • the deterministic selection asks no model and names what it left out
    • without a model the run needs review unless something fails
    • status mapping, and the overall status of none
  • MCP-01 coverage row.

Integration over MCP, each on SQLite and PostgreSQL:

  • S2: a draft with "best in class", an image without alt text and an unsourced price fails three checks with their findings. The fixed draft with a source passes, and nothing is stored.
  • An active LLM-judged check needs review without a model, and the run does too. With deterministic it is listed under skipped and the run passes.
  • A draft of an update is checked for what it changes: a headline edit passes and a price change without a source fails.
  • A draft the content type refuses lists its field errors and fails.
  • Unknown content types, entries, arguments and selections are refused with what to change, and a type outside the agent's scope is refused.
  • The built-in tool list includes check_draft.

The full suite ran locally against a postgres:17 container as well as SQLite, and markdownlint passes. No new package and no migration.

Checklist

  • Linked to an issue a maintainer has confirmed
  • Targets v2
  • Tests or docs updated where behaviour changed
  • No tool-generated co-author trailers or footers

@pournasserian pournasserian added this to the M1 Governance core milestone Oct 19, 2026
@pournasserian pournasserian added type:feature New capability v2 FluentCMS v2 (v2 branch) area:checks labels Oct 19, 2026
@pournasserian
pournasserian merged commit 955d984 into v2 Oct 19, 2026
2 checks passed
@pournasserian
pournasserian deleted the 2554-check-draft branch October 19, 2026 02:51
pournasserian added a commit that referenced this pull request Oct 19, 2026
Closes #2573

## What

The check run slice of F-CHK (#2420), CHK-03 and PERF-02: every
submitted change set is checked by the check engine. The results are
stored with the set, and the engine, not the agent, moves the set to
checks failed or awaiting review. It runs as `SystemInitiator.Checks`
(#2558, ADR-0012) with the runner from #2556 and the job queue from
#2536.

- **Core** (`FluentCMS.Core.Checks`):
- `IChangeSetChecks.SubmitAsync` moves a set to checks running and
enqueues a `checks.run` job, both in one unit of work, so a set in
checks running always has a run coming.
- While every active check is deterministic, a set of up to 50
operations is checked on the submitting request, in a scope acting as
the engine. Its job waits two minutes and does nothing if that run
finished. Any other set is checked by the job, which reports progress at
most once a second.
- `RunAsync` checks each entry and entity the set creates or updates at
its proposed version, against the content type version it was written
to. An update's changed fields come from its base version. The active
checks are read once, so every item is judged by the same checks.
- An LLM-judged check's model call is charged to the agent that opened
the set (`CheckInput.ChargedTo`, SEC-04), not to the engine.
- The run, its results, `change_set.checked`, one `check.judged` per
LLM-judged result (model and prompt hash, CHK-05) and the move all
commit together.
- **Runs belong to submissions.** A run stores its submission's job id.
It does nothing unless the set is in checks running and that job is the
set's latest `checks.run` job. This is checked again after the move,
inside the unit of work, so a run of an earlier submission, or one
another run beat, rolls back and changes nothing. The job payload names
only the set.
- **Sets do not stay in checks running.** An item that cannot be read,
such as an entry deleted after it was proposed, is a problem that fails
the run, and the other items are still checked. A conflict is retried. A
job that still throws on its last attempt (6) fails the set with a
problem. A set whose job ended without moving it, as when the host stops
during the last attempt, is failed when the agent submits it again, so
it can be reopened.
- **A failed set is fixed in place.** A set holds one operation per
item. Until now, an agent that reopened a failed set could not change an
entry already in it, because the update was refused as a duplicate. Now
a later draft of an entry or entity the set already proposes moves that
operation on to the new draft. It must be the same change from the same
published version. The item keeps one operation, one diff and one
version to restore, and `change_set.operation_updated` records the move.
Other duplicates, including content type operations, are still refused.
The tools' size pre-check no longer counts such a fix as a new
operation.
- **Reverts are not held.** A revert takes no new operations, so it
could never pass a check it fails, such as one asking for a source. Its
run is recorded and it goes to review with the results (CHG-06).
- **Infrastructure**: tables `CheckRuns` (numbered per set, unique on
set and number, with problems as JSON) and `CheckResults` (one row per
item and check, with findings and the model call). Migrations for SQLite
and PostgreSQL.
- **Engine scopes** (`EngineScopes.AddEngineScopes`): a host wraps its
`ICurrentInitiator` so Core can open a scope acting as an engine for the
inline run. The scope is opened before the set moves, so a host that
cannot run the engine fails the submit rather than leaving every set to
its job. Opening one is internal to Core and takes only a
`SystemInitiator`, so no other code can act as someone it is not. The
job worker keeps its own wrapper.
- **Tools**:
- `submit_change_set` returns the run when it ran inline, or the job to
follow otherwise.
- New read-only `get_change_set_checks` returns the run of the latest
submission (none while a resubmitted set is still running) and its job's
state and progress, without the raw error a worker recorded.
  - MCP-01 coverage row "Read the check results of a change set".
- `ICheckRunner` gains an overload that takes the active checks.
- CONTRIBUTING.md: the check run.

## Why

CHK-03 requires checks on every change set before routing, with results,
evidence and cost stored with the set. PERF-02 requires deterministic
checks within ten seconds for 200 items, with LLM-judged checks running
asynchronously with progress.

Choices worth a look:

- **Inline for small sets while every check is deterministic, a job
otherwise.** The agent gets its findings in the answer to
`submit_change_set` for the common case. The job enqueued with every
submission makes the run durable even if the inline run never finishes.
While any LLM-judged check is active, every set runs as a job, since a
model call never holds up a request. Deciding by the checks whose scope
applies would need the set's items read before the submit; that can come
later if jobs prove slow.
- **The inline run ignores the request's cancellation.** The set is
already submitted, so an agent that disconnects still gets its set
checked, and an inline failure or fault is left to the job instead of
being reported as a failed submit.
- **Problems as a run-level list**, not fake check results: they say
what kept the engine from checking an item and what to do, and they fail
the run.
- **Fixing in place, in this PR.** It is a change to the change set
model (F-CHG), made here because this PR is the first to leave sets in
checks failed. The F-CHG lane knows.
- **Core logs nothing**, so an inline fault is not logged; the job
checks the set two minutes later and records the error if the fault
recurs.
- **Plugin check types run in the engine's scope.** They are compiled-in
extensions today. Per-plugin permissions come with the plugin SDK
(EXT-03).
- **Passing sets wait for review.** Routing (F-RTE #2421) decides more
once it lands.
- **Not in this PR**: checks that read entities (F-ENT #2417), and a
scheduled retention purge as `SystemInitiator.Retention`.

## Tests

Unit:

- `CheckRunJob`:
  - it runs as the engine for its own submission and reports progress
  - an invalid payload fails the job
  - a conflict is retried
- a fault before the last attempt is retried, and on the last attempt it
fails the set
  - a lost lease ends the attempt without failing the set
- `CheckRunner`: an LLM-judged check charges its model call to the agent
named.
- `ChangeSet`: a later draft of an entry, as a create or an update from
the same published version, moves its operation on. A draft that is not
later, or a content type operation, is still a duplicate.
- MCP-01 coverage row.

Integration over MCP, on SQLite and PostgreSQL unless noted:

- S2: a set with "best in class", an image without alt text and
unsourced copy is checked inline and lands in checks failed, with three
failing results whose findings point at the field.
`get_change_set_checks` reads the run back, along with its waiting
backup job. The agent reopens the set, updates the entry from the draft
the set proposes with a source, and submits it again. The same set
passes to awaiting review as run 2, holding one create operation at the
new draft. The engine's four events name the check engine, with no
credential, and `change_set.operation_updated` records the fix.
- A set no check applies to passes to awaiting review.
- The job of an earlier submission changes nothing, both while the set
is reopened and after it is submitted again, even when a person runs it.
- An item that cannot be read fails the set with a problem naming it,
and the other item is still checked.
- While an LLM-judged plugin check is active, a set is checked as a job
with progress. The check runs as `SystemInitiator.Checks` and is charged
to the agent's credential. Its `check.judged` event names the model, the
prompt hash and the set, as the engine, and the run's cost is the
judgement's.
- A revert whose restored version fails the sources check goes to
awaiting review with the failing result.
- A set whose check job died while it was in checks running is failed
when its agent submits it again, with a problem naming the job, and can
be reopened. A person submitting a set whose job failed fails it the
same way.
- PERF-02: deterministic checks on a 200-item set run as a job and
finish in under ten seconds.
- The existing write tool test now expects a submitted schema-only set
to pass to awaiting review.
- A flaky activation race test (`CheckDirectoryTests`) accepts either
refusal the second activation can legitimately get.

The full suite ran locally against a `postgres:17` container as well as
SQLite, and markdownlint passes. No new package. This PR adds no
`IContentTypeDirectory.CreateAsync` or `ReviseAsync` call.

## Checklist

- [x] Linked to an issue a maintainer has confirmed
- [x] Targets `v2`
- [x] Tests or docs updated where behaviour changed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:feature New capability v2 FluentCMS v2 (v2 branch)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pre-flight check of a draft without opening a change set

1 participant