Skip to content

Check run on every change set, with results stored on the set #2573

Closed

Description

@pournasserian

Part of #2420 (F-CHK). The check run on every change set (CHK-03, PERF-02): a submitted set is checked by the check engine, the results are stored with the set, and the engine moves it to checks failed or awaiting review. It runs as SystemInitiator.Checks (#2558, ADR-0012), uses the runner from #2556 and the job queue from #2536.

Scope

  • Core (FluentCMS.Core.Checks):
    • IChangeSetChecks.SubmitAsync moves a set to checks running through IChangeSetDirectory.TransitionAsync and enqueues a checks.run job in the same unit of work.
    • While every active check is deterministic, a set of up to 50 operations is checked on the submitting request, as the check engine. The job waits two minutes and only runs if that run did not finish. Any other set is checked by the job, which reports progress.
    • Each entry and entity the set creates or updates is checked at its proposed version, against the content type version it was written to. An update's changed fields come from its base version.
    • An LLM-judged check's model call is charged to the agent that opened the set (SEC-04), not to the engine.
    • The run, its results and the move commit together: checks failed when a check fails, otherwise awaiting review (routing comes with F-RTE). Audit events: change_set.checked, and check.judged for each LLM-judged result with its model and prompt hash (CHK-05).
    • Each run belongs to its submission's job. The job re-reads the set and does nothing unless it is in checks running for that submission; the payload names only the set.
    • A reopened set is fixed in place: a later draft of an entry or entity it already proposes moves that operation on.
    • An item that cannot be read, such as an entry deleted since it was proposed, is a problem that fails the run. A job whose last attempt throws fails the set too, and a set whose job ended without moving it is failed when it is submitted again.
    • A revert goes to review with its check results, since it takes no new operations and could never pass a check it fails (CHG-06).
  • Infrastructure: tables CheckRuns (numbered per set) and CheckResults (one row per item and check, with findings and model call), with migrations for SQLite and PostgreSQL.
  • Engine scopes: Core opens a scope acting as an engine for the inline run (AddEngineScopes). Only Core can, and only as an engine.
  • Tools:
    • submit_change_set returns the run when it was checked inline, or the job to follow otherwise.
    • New read-only get_change_set_checks returns the run of the latest submission and its job.
    • MCP-01 coverage row "Read the check results of a change set".
  • CONTRIBUTING.md: the check run.

Out of scope

Acceptance

  • S2 over MCP: an agent submits a product page with "best in class", an image without alt text and an unsourced claim. The set is in checks failed, with three failing results that say what to change, and the move is logged as the check engine, not the agent. The agent reopens the set and fixes the entry in it, and the same set passes to awaiting review.
  • A set no check applies to passes to awaiting review.
  • While an LLM-judged check is active, a set is checked as a job with progress, as the check engine, and the model call is charged to the agent.
  • PERF-02: deterministic checks on a 200-item set finish in under ten seconds.
  • An agent still cannot move a set out of checks running itself.
  • Everything above passes on SQLite and PostgreSQL. No new package.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type:featureNew capabilityv2FluentCMS v2 (v2 branch)

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions