Run the checks on every submitted change set

CONTRIBUTING.md

@@ -138,6 +138,8 @@ An LLM-judged type asks a model only through `IModelGateway`, which it reads fro
 
 A check is a versioned definition under a slug: a check type and its settings, a description of what it enforces, a severity (block or warn), an optional scope of content types and fields, and test examples, at least one it must pass and one it must fail. `ICheckDirectory` activates a version only after running its examples, and names each example that does not get its verdict (CHK-02). A revision leaves the active version in place until the new one is activated, and only a person activates or deactivates a check. `ICheckRunner` runs the active checks that apply to a piece of content and gives each one's status: pass, warn for a failing check of warn severity, fail, or needs review. Agents run it on a draft with the `check_draft` tool before they propose it (CHK-04).
 
+A submitted change set is checked as the check engine (`SystemInitiator.Checks`, ADR-0012), never as the agent: submit it through `IChangeSetChecks.SubmitAsync`, not by moving it to checks running yourself. While every active check is deterministic, a set of up to 50 operations is checked on the request that submits it; any other set is checked by a `checks.run` job, which reports its progress (PERF-02). Every entry and entity the set creates or updates is checked at the version it proposes, and an LLM-judged check charges its model call to the agent that opened the set (SEC-04). An item the engine cannot read, such as an entry deleted since, is a problem that fails the run. So is a job whose last attempt throws, and a set whose job ended without moving it is failed when it is submitted again, so it can be reopened. A reopened set is fixed in place: a set holds one operation per item, and a later draft of an entry or entity it already proposes, the same change from the same published version, moves that operation on to the new draft and appends `change_set.operation_updated`. Each run names its submission's job, and a run of an earlier submission changes nothing. A revert is never held by its checks: it goes to review with their results (CHG-06). The run, each item's results with their findings and model calls, and the move it decides (checks failed when a check fails, otherwise awaiting review) commit together, numbered per set, with the audit events `change_set.checked` and `check.judged` (CHK-03, CHK-05). Agents read the run of the latest submission and its job with the `get_change_set_checks` tool, which shows a job's state and progress but not the error a worker recorded.
+
 Search (`ISearchIndex` in `FluentCMS.Core.Search`, SRC-01, ADR-0010) indexes each entry's latest version in the same unit of work as the write, as the lowercase words of every string and number in its fields (`SearchText`), so a result appears the moment its write commits. PostgreSQL keeps them in a `tsvector` with a GIN index and SQLite in an FTS5 table, both outside the EF model and created by their migrations; an external engine implements the same interface, registered with `AddFluentCmsExtension<ISearchIndex, T>(ServiceLifetime.Scoped)` when it needs the request's scope. `ISearch` combines the matches with the entry rows, so deleting, restoring and publishing change results without touching the index, follows a match to the entries that reference it, and returns only types the caller may read. It reads the matches a page at a time, `Search__PageSize` (1,000 by default), and drops what the scope, deletions and filters leave out, until enough are left, the index has no more, or it has read `Search__MaxMatches` (10,000). A result that stopped there says it is not complete. Each hit has the state the entry tools show, in review included. A string that is a GUID, such as a reference, is not indexed.
 
 ## Code style

src/FluentCMS.Core/Builtin/Checks/ModelJudgement.cs

@@ -72,7 +72,7 @@ public static async Task<CheckOutcome> JudgeAsync(
         data.AddRange(settings);
 
         var asked = await gateway.AskAsync(
-            new ModelRequest(Profile, $"check:{checkType}", task + "\n\n" + AnswerRules, data) { AnswerSchema = AnswerSchema },
+            new ModelRequest(Profile, $"check:{checkType}", task + "\n\n" + AnswerRules, data) { AnswerSchema = AnswerSchema, ChargedTo = input.ChargedTo },
             cancellationToken);
         if (!asked.IsSuccess)
         {

src/FluentCMS.Core/Builtin/Jobs/CheckRunJob.cs

@@ -0,0 +1,56 @@
+using System.Text.Json;
+using FluentCMS.Core.ChangeSets;
+using FluentCMS.Core.Checks;
+using FluentCMS.Core.Identity;
+using FluentCMS.Core.Jobs;
+using FluentCMS.Core.Results;
+using Microsoft.Extensions.DependencyInjection;
+
+namespace FluentCMS.Core.Builtin.Jobs;
+
+// The check run of a submitted change set (CHK-03), as the check engine whoever submitted the set (ADR-0012). A run is safe to
+// repeat: a set the checks have already moved on is left as it is. The payload only names the set; whether to run, and where the
+// set goes, come from what is stored.
+internal sealed class CheckRunJob : IJobHandler
+{
+    public string Type => ChangeSetChecks.JobType;
+
+    public SystemInitiator RunAs => SystemInitiator.Checks;
+
+    public async Task<Result> RunAsync(JobContext context, CancellationToken cancellationToken)
+    {
+        var payload = context.Job.Payload;
+        if (payload.ValueKind != JsonValueKind.Object
+            || !payload.TryGetProperty("changeSetId", out var id)
+            || id.ValueKind != JsonValueKind.String
+            || !id.TryGetGuid(out var changeSetId))
+        {
+            return Result.Failure("payload_invalid", "A check run names the change set it checks as changeSetId.");
+        }
+
+        var checks = context.Services.GetRequiredService<IChangeSetChecks>();
+        try
+        {
+            var ran = await checks.RunAsync(
+                changeSetId,
+                context.Job.Id,
+                CheckRunMode.Job,
+                (done, total) => context.ReportProgressAsync(done, total, $"Checked {done} of {total} items.", cancellationToken),
+                cancellationToken);
+
+            // A write that met the run's own can be got past by another attempt on what is stored then.
+            if (!ran.IsSuccess && ran.Errors.Any(error => error.Code == ChangeSetErrorCodes.Conflict))
+            {
+                throw new InvalidOperationException(ran.Errors[0].Message);
+            }
+
+            return ran.IsSuccess ? Result.Success() : Result.Failure(ran.Errors);
+        }
+        catch (Exception exception) when (exception is not OperationCanceledException && context.Job.Attempts >= context.Job.MaxAttempts)
+        {
+            // The last attempt fails the set, so the agent can reopen it, rather than leave it in checks running for good.
+            var abandoned = await checks.AbandonAsync(changeSetId, context.Job.Id, CheckRunMode.Job, cancellationToken);
+            return Result.Failure(abandoned.IsSuccess ? [new Error("check_run_abandoned", exception.Message)] : abandoned.Errors);
+        }
+    }
+}

src/FluentCMS.Core/Builtin/Tools/ChangeSetTarget.cs

@@ -84,7 +84,8 @@ changeSetId is not null
 
     // Refuses ahead of the write what the change set would refuse after it, so no version is stored for nothing and a refused
     // cap keeps its security event (SEC-08), which a rolled-back write would take with it. The set checks again on the add.
-    public async Task<Result> CheckAsync(IServiceProvider services, CancellationToken cancellationToken)
+    // An item the set already holds takes no more room, since its operation moves on to the newer version.
+    public async Task<Result> CheckAsync(IServiceProvider services, Guid? itemId, CancellationToken cancellationToken)
     {
         var directory = services.GetRequiredService<IChangeSetDirectory>();
         var permissions = services.GetRequiredService<IAgentPermissions>();
@@ -95,7 +96,8 @@ public async Task<Result> CheckAsync(IServiceProvider services, CancellationToke
 
         if (set.IsSuccess)
         {
-            return await permissions.CheckChangeSetSizeAsync(set.Value, adding: 1, cancellationToken);
+            var adding = itemId is { } item && set.Value.Operations.Any(operation => operation.ItemId == item) ? 0 : 1;
+            return await permissions.CheckChangeSetSizeAsync(set.Value, adding, cancellationToken);
         }
 
         return _changeSetId is null ? await permissions.CheckNewChangeSetAsync(cancellationToken) : Result.Failure(set.Errors);

src/FluentCMS.Core/Builtin/Tools/ChangeSetTools.cs

@@ -1,5 +1,6 @@
 using System.Text.Json.Nodes;
 using FluentCMS.Core.ChangeSets;
+using FluentCMS.Core.Checks;
 using FluentCMS.Core.Results;
 using FluentCMS.Core.Tools;
 using Microsoft.Extensions.DependencyInjection;
@@ -47,20 +48,23 @@ public sealed class ChangeSetTools : IToolProvider
         new AgentTool(
             "submit_change_set",
             "Submit a change set for checks and review",
-            "Submits one of your open change sets once it holds every operation it needs: checks run on it, and a person reviews "
-            + "it before anything is published. A submitted set takes no more operations. An agent never approves or publishes "
-            + "a set; if checks fail, reopen it with reopen_change_set, fix it and submit it again.",
+            "Submits one of your open change sets once it holds every operation it needs: the site's checks run on it, and a person "
+            + "reviews it before anything is published. A small set is checked at once and the answer holds its check results; a "
+            + "larger one, or any set while an LLM-judged check is active, is checked in the background, so follow it with get_change_set_checks. "
+            + "A submitted set takes no more operations. An agent never approves or publishes a set; if checks fail, reopen it with "
+            + "reopen_change_set, fix what they found and submit it again.",
             ToolSchemas.Object(ToolSchemas.Uuid("id", "The id of your open change set.")),
             readOnly: false,
-            (call, cancellationToken) => MoveAsync(call, ChangeSetState.ChecksRunning, cancellationToken)),
+            SubmitAsync),
         new AgentTool(
             "reopen_change_set",
             "Reopen a change set after failed checks",
-            "Reopens one of your change sets whose checks failed, so you can add operations that fix it and submit it again. "
-            + "Read the set with get_change_set first.",
+            "Reopens one of your change sets whose checks failed, so you can fix it and submit it again: update an item from the "
+            + "version the set proposes, with update_entry and the set's change_set_id, or add operations. Read what failed with "
+            + "get_change_set_checks first.",
             ToolSchemas.Object(ToolSchemas.Uuid("id", "The id of your change set whose checks failed.")),
             readOnly: false,
-            (call, cancellationToken) => MoveAsync(call, ChangeSetState.Open, cancellationToken)),
+            ReopenAsync),
     ];
 
     private static async Task<ToolResult> OpenAsync(ToolInvocation call, CancellationToken cancellationToken)
@@ -81,7 +85,7 @@ private static async Task<ToolResult> OpenAsync(ToolInvocation call, Cancellatio
         });
     }
 
-    private static async Task<ToolResult> MoveAsync(ToolInvocation call, ChangeSetState target, CancellationToken cancellationToken)
+    private static async Task<ToolResult> SubmitAsync(ToolInvocation call, CancellationToken cancellationToken)
     {
         var arguments = new ToolArgumentsReader(call.Arguments);
         var id = arguments.RequiredGuid("id");
@@ -90,15 +94,46 @@ private static async Task<ToolResult> MoveAsync(ToolInvocation call, ChangeSetSt
             return ToolResult.Failure(errors);
         }
 
-        var set = await call.Services.GetRequiredService<IChangeSetDirectory>().TransitionAsync(id!.Value, target, cancellationToken);
+        var submitted = await call.Services.GetRequiredService<IChangeSetChecks>().SubmitAsync(id!.Value, cancellationToken);
+        if (!submitted.IsSuccess)
+        {
+            return ToolResult.Failure(submitted.Errors, Hint(Result<ChangeSet>.Failure(submitted.Errors)));
+        }
+
+        var submission = submitted.Value;
+        var set = submission.ChangeSet;
+        return ToolResult.Success(new JsonObject
+        {
+            ["changeSet"] = ToolJson.ChangeSet(set),
+            ["checks"] = submission.Run is { } run ? ToolJson.CheckRun(run) : null,
+            ["job"] = submission.Run is null ? ToolJson.Job(submission.Job) : null,
+            ["next"] = set.State switch
+            {
+                ChangeSetState.ChecksFailed =>
+                    "Checks failed: each failing result and problem says what to change. Call reopen_change_set, update the items from the "
+                    + "versions the set proposes and submit the set again.",
+                ChangeSetState.AwaitingReview => "Checks passed, and a person reviews the set next. Follow its state with get_change_set.",
+                _ => $"Checks run in the background as job {submission.Job.Id}. Follow them with get_change_set_checks.",
+            },
+        });
+    }
+
+    private static async Task<ToolResult> ReopenAsync(ToolInvocation call, CancellationToken cancellationToken)
+    {
+        var arguments = new ToolArgumentsReader(call.Arguments);
+        var id = arguments.RequiredGuid("id");
+        if (arguments.Errors() is { Count: > 0 } errors)
+        {
+            return ToolResult.Failure(errors);
+        }
+
+        var set = await call.Services.GetRequiredService<IChangeSetDirectory>().TransitionAsync(id!.Value, ChangeSetState.Open, cancellationToken);
         return ToolResult.From(
             set,
             moved => new JsonObject
             {
                 ["changeSet"] = ToolJson.ChangeSet(moved),
-                ["next"] = target == ChangeSetState.Open
-                    ? "The set is open again: add the operations that fix it, then call submit_change_set."
-                    : "Checks run next, and a person reviews the set. Follow its state with get_change_set.",
+                ["next"] = "The set is open again: update its items from the versions it proposes, or add operations, then call submit_change_set.",
             },
             Hint(set));
     }

src/FluentCMS.Core/Builtin/Tools/CheckTools.cs

@@ -44,6 +44,15 @@ public sealed class CheckTools : IToolProvider
             // Not read-only: it stores nothing, but an LLM-judged check spends the agent's model budget.
             readOnly: false,
             CheckDraftAsync),
+        new AgentTool(
+            "get_change_set_checks",
+            "Get a change set's check results",
+            "Returns the latest check run of a change set (CHK-03): its status, cost and each item's results, with findings that say "
+            + "what to change and where, plus the background job that runs the checks with its progress. A set's checks run when it "
+            + "is submitted; if they failed, call reopen_change_set, fix the items and submit the set again.",
+            ToolSchemas.Object(ToolSchemas.Uuid("id", "The change set's id.")),
+            readOnly: true,
+            ChangeSetChecksAsync),
     ];
 
     private static async Task<ToolResult> CheckDraftAsync(ToolInvocation call, CancellationToken cancellationToken)
@@ -96,13 +105,13 @@ [new Error(ToolErrorCodes.ArgumentInvalid, $"The entry {entryId} is of the conte
         var run = await services.GetRequiredService<ICheckRunner>().RunAsync(
             new CheckSubject(fields.Value, sources, slug, definition, changed),
             checks == "deterministic" ? CheckSelection.Deterministic : CheckSelection.All,
-            cancellationToken);
+            cancellationToken: cancellationToken);
 
         // A draft the content type refuses cannot be proposed, whatever the checks say.
         return ToolResult.Success(new JsonObject
         {
             ["status"] = ToolJson.Name(refused.Count > 0 ? CheckStatus.Fail : CheckResult.Overall(run.Results)),
-            ["results"] = new JsonArray([.. run.Results.Select(Result)]),
+            ["results"] = new JsonArray([.. run.Results.Select(ToolJson.CheckResult)]),
             ["skipped"] = new JsonArray([.. run.Skipped.Select(name => JsonValue.Create(name))]),
             ["fieldErrors"] = new JsonArray([.. refused.Select(error => new JsonObject { ["code"] = error.Code, ["message"] = error.Message })]),
         });
@@ -119,26 +128,21 @@ private static List<string> Changed(JsonElement draft, JsonElement latest)
             .ToList();
     }
 
-    private static JsonObject Result(CheckResult result)
+    private static async Task<ToolResult> ChangeSetChecksAsync(ToolInvocation call, CancellationToken cancellationToken)
     {
-        return new JsonObject
+        var arguments = new ToolArgumentsReader(call.Arguments);
+        var id = arguments.RequiredGuid("id");
+        if (arguments.Errors() is { Count: > 0 } errors)
         {
-            ["check"] = result.Check,
-            ["version"] = result.Version,
-            ["type"] = result.Type,
-            ["kind"] = ToolJson.Name(result.Kind),
-            ["severity"] = ToolJson.Name(result.Severity),
-            ["description"] = result.Description,
-            ["status"] = ToolJson.Name(result.Status),
-            ["findings"] = new JsonArray([.. result.Findings.Select(finding => new JsonObject
-            {
-                ["message"] = finding.Message,
-                ["pointer"] = finding.Pointer,
-                ["excerpt"] = finding.Excerpt,
-            })]),
-            ["model"] = result.Model is { } model
-                ? new JsonObject { ["callId"] = model.CallId.ToString(), ["provider"] = model.Provider, ["model"] = model.Model, ["cost"] = model.Cost }
-                : null,
-        };
+            return ToolResult.Failure(errors);
+        }
+
+        var state = await call.Services.GetRequiredService<IChangeSetChecks>().GetAsync(id!.Value, cancellationToken);
+        return ToolResult.From(state, read => new JsonObject
+        {
+            ["changeSet"] = ToolJson.ChangeSet(read.ChangeSet),
+            ["checks"] = read.Run is { } run ? ToolJson.CheckRun(run) : null,
+            ["job"] = read.Job is { } job ? ToolJson.Job(job) : null,
+        });
     }
 }

src/FluentCMS.Core/Builtin/Tools/EntityTools.cs

@@ -233,7 +233,8 @@ private static async Task<ToolResult> SetFactAsync(ToolInvocation call, Cancella
                 return Result.Success();
             }, cancellationToken),
             ProposedEntryWrite.Update,
-            cancellationToken);
+            cancellationToken,
+            id);
     }
 
     private static async Task<ToolResult> RelateAsync(ToolInvocation call, CancellationToken cancellationToken)
@@ -290,7 +291,8 @@ private static async Task<ToolResult> RelateAsync(ToolInvocation call, Cancellat
                 return Result.Success();
             }, cancellationToken),
             ProposedEntryWrite.Update,
-            cancellationToken);
+            cancellationToken,
+            id);
     }
 
     // Reads the entity at the version the agent read, changes its fields in a copy and writes the next version; a stale base version

src/FluentCMS.Core/Builtin/Tools/EntryTools.cs

@@ -242,7 +242,8 @@ private static async Task<ToolResult> UpdateAsync(ToolInvocation call, Cancellat
             target!,
             entries => entries.UpdateAsync(id!.Value, baseVersion!.Value, fields!.Value, cancellationToken),
             ProposedEntryWrite.Update,
-            cancellationToken);
+            cancellationToken,
+            id);
     }
 
     private static async Task<ToolResult> DeleteAsync(ToolInvocation call, CancellationToken cancellationToken)

src/FluentCMS.Core/Builtin/Tools/ProposedEntryWrite.cs

@@ -23,10 +23,11 @@ public static async Task<ToolResult> RunAsync(
         ChangeSetTarget target,
         Func<IEntryDirectory, Task<Result<Entry>>> write,
         Func<Entry, ItemKind, Result<ChangeOperation>> operation,
-        CancellationToken cancellationToken)
+        CancellationToken cancellationToken,
+        Guid? itemId = null)
     {
         var services = call.Services;
-        var ready = await target.CheckAsync(services, cancellationToken);
+        var ready = await target.CheckAsync(services, itemId, cancellationToken);
         if (!ready.IsSuccess)
         {
             return ToolResult.Failure(ready.Errors);