feat(testing): Tests areas stream per-case progress, bound every case, and get a console (memex tests) - #5755
Conversation
…ase, and gets a console A node's Tests area ran every case and rendered ONE frame at the end, so a case that called an external service live (or hung) left the page on "Rendering …" for the whole suite, with nothing to tell slow from stuck and nothing naming the case. - MeshTestRunner streams: every case pending at once, the running case with its ticking elapsed time and its output lines as written, finished cases ✔/✖ — then ONE verdict frame (✅/❌, "N/M passed"). Progress frames carry AreaFrameClassifier.TestsRunningId and never the verdict glyphs, so no consumer can read one as a verdict. - MeshTestCase + Area(host, suite, cases): the listed-cases shape (static methods) runs through the same streaming runner; synchronous bodies run on the Tests I/O pool, so a body that never returns fails "timed out: no verdict within Ns" instead of freezing the render. - AreaProbe (the plugin gate) treats a progress frame as transient; a cut-off run names its progress. - memex tests <path>: reads the area until the verdict, printing each case as it changes. - Column titles and progress title localized (en/de). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Critical CLI polling and per-case output attribution issues can prevent verdicts and misreport results.
Review effort: Lite
Findings: 4
Open (13)
What changed in this PR
Adds streamed Tests-area progress, bounded test cases, gate handling, localization, and a memex tests console command.
Changes:
- Streams pending, running, output, and final verdict frames.
- Adds listed synchronous/live test-case APIs.
- Updates CLI support, tests, localization, and documentation.
| File | Description |
|---|---|
tools/MeshWeaver.PluginTester/AreaProbe.cs |
Treats progress frames as transient |
test/MeshWeaver.Testing.InMesh.Test/MeshTestRunnerTests.cs |
Tests runner progress behavior |
test/MeshWeaver.PluginTester.Test/AreaProbeTest.cs |
Tests gate classification |
test/MeshWeaver.Cli.Test/TestsCommandTest.cs |
Tests CLI parsing and output |
src/MeshWeaver.Testing.InMesh/MeshTestRunner.Progress.cs |
Implements progress snapshots and rendering |
src/MeshWeaver.Testing.InMesh/MeshTestRunner.cs |
Integrates streaming execution |
src/MeshWeaver.Testing.InMesh/MeshTestCase.cs |
Adds listed test-case definitions |
src/MeshWeaver.Messaging.Hub/Localization/strings.en.json |
Adds English labels |
src/MeshWeaver.Messaging.Hub/Localization/strings.de.json |
Adds German labels |
src/MeshWeaver.Layout/AreaFrameClassifier.cs |
Classifies Tests progress frames |
src/MeshWeaver.Documentation/Data/Architecture/WritingTests.md |
Documents streaming Tests areas |
src/MeshWeaver.Documentation/Data/Architecture/DecentralisedTests.md |
Updates gate documentation |
src/MeshWeaver.Cli/TestsCommand.cs |
Adds console polling and parsing |
src/MeshWeaver.Cli/Program.cs |
Registers the tests command |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| /// <summary>One case as the area renders it.</summary> | ||
| /// <param name="Case">The case's name.</param> | ||
| /// <param name="Result">Its status glyph / verdict.</param> | ||
| /// <param name="Time">Its elapsed time.</param> | ||
| /// <param name="Output">Its failure message and output lines.</param> | ||
| public sealed record Row(string Case, string Result, string Time, string Output); |
There was a problem hiding this comment.
Fixed in 069e63f. The CLI no longer tracks rows; it prints the run's activity lines. The reader that now tracks rows (TestsAreaFrame, used by RunTests) keys by class + case (Row.Key). TestsAreaFrameTest pins two classes' same-named cases staying apart.
| string? lastTitle = null; | ||
| while (true) | ||
| { | ||
| var body = await client.Get(area, ct); |
There was a problem hiding this comment.
Agreed, and measured on memex-cloud: two get reads 34 s apart each filed a new pair of the Maintenance suite's live request nodes, and the second read answered with an earlier subscription's cached verdict. In 069e63f, MeshOperations.RunTests holds ONE area subscription for the whole run and writes it to an activity. memex tests now POSTs api/mesh/run-tests and polls the ACTIVITY, which starts nothing. Pinned end to end by RunTestsStreamsIntoAnActivityTest.
| var finished = run(() => signals.OnNext(new StartedEvent()), line => signals.OnNext(new LineEvent(line))) | ||
| .Select(result => (RunEvent)new FinishedEvent(result)); |
There was a problem hiding this comment.
Fixed in 069e63f. A listed case's writer is closed at its verdict (OutputLines.Close), so a body that outlives its bound cannot write onto the next row. In the fold, a line arriving with no running case is dropped.
| // Every line a case writes lands in its verdict's detail AND streams to the progress | ||
| // frame the moment it is written — a slow case shows what it is doing while it does it. | ||
| var context = host is null ? null : new MeshTestContext(host, partition, line => { output.Add(line); onLine(line); }, deadline); |
There was a problem hiding this comment.
Fixed in 069e63f. The class-wide writer now admits only the RUNNING case's lines. Each case enters a token (AsyncLocal, which flows into its continuations) inside its pool leaf, and the gate is closed at the case's terminal. A leaked case's late lines carry its own token and are dropped instead of landing in the next case's detail or row.
| string? lastTitle = null; | ||
| while (true) | ||
| { | ||
| var body = await client.Get(area, ct); |
There was a problem hiding this comment.
Fixed in 069e63f. --timeout is now a linked CancellationTokenSource with CancelAfter, passed to every request (the run-tests POST and each activity read) and to the delay. An elapsed deadline maps to exit 4, and the user's own Ctrl-C is kept distinct.
| /// <param name="interval">How often to read the area.</param> | ||
| /// <param name="output">Where the console goes.</param> | ||
| /// <param name="ct">Cancels the wait.</param> | ||
| public static async Task<int> Run(MemexClient client, string path, TimeSpan timeout, TimeSpan interval, TextWriter output, CancellationToken ct) |
There was a problem hiding this comment.
Same answer as on Program.cs: this is the CLI executable's boundary, an HTTP client process with no hub or turn scheduler. It matches the existing async Run helper that every memex verb goes through.
| public static async Task<int> Run(MemexClient client, string path, TimeSpan timeout, TimeSpan interval, TextWriter output, CancellationToken ct) | ||
| { | ||
| var area = $"@{path.TrimStart('@').TrimEnd('/')}/area/Tests"; | ||
| var printed = new Dictionary<string, Row>(StringComparer.Ordinal); |
There was a problem hiding this comment.
Fixed in 069e63f. The CLI now carries only an int (lines printed). The row-change state lives in TestsAreaFrame.Changes, which takes and returns an ImmutableDictionary.
| var body = await client.Get(area, ct); | ||
| if (body.StartsWith("Error:", StringComparison.Ordinal) || body.StartsWith("Not found", StringComparison.Ordinal)) | ||
| { | ||
| await output.WriteLineAsync(body); |
There was a problem hiding this comment.
Same as the two async threads above: the awaits are in the console executable (HTTP client, no mesh scheduler), consistent with the existing Run helper in Program.cs.
| CaseResult Verdict(string? failure) => failure is null | ||
| ? new CaseResult(suite, c.Name, "✅ pass", output.Joined, DateTimeOffset.UtcNow - started) | ||
| : new CaseResult(suite, c.Name, "❌ FAIL", failure + (output.Joined.Length > 0 ? " · " + output.Joined : ""), DateTimeOffset.UtcNow - started); |
There was a problem hiding this comment.
Fixed in 069e63f. The rendered verdict word beside each glyph is localized: tests.result.passed / failed / skipped, en and de, resolved once in the render scope (ColumnTitles.Display). The stored CaseResult keeps the glyph, which is the contract every reader (gate, RunTests, tests) matches. The verdict TITLE stays English on purpose, because its 'N/M passed' is the plugin gate's parse contract.
| .WithId(AreaFrameClassifier.TestsRunningId); | ||
|
|
||
| private static CaseRow Row(CaseResult r, string result, string detail) => | ||
| new(r.Class, r.Name, result, IsUnfinished(r) && r.Result == PendingResult ? "" : $"{r.Elapsed.TotalSeconds:0.0}s", detail); |
There was a problem hiding this comment.
Fixed in 069e63f. Elapsed is now formatted with CultureInfo.InvariantCulture, and the progress title already was.
…on, sync-body leaks, i18n, culture) - MeshOperations.RunTests(path): holds ONE subscription to the node's Tests area and writes it to an activity in the caller's partition (category TestRun) — a line per case, its output, and a terminal status that is the verdict. Polling get @node/area/Tests re-runs the suite on every read (measured on memex-cloud: each read filed the Maintenance suite's live request nodes again and answered with an earlier subscription's cached verdict). - TestsAreaFrame (MeshWeaver.Layout): the pure reader of a Tests area frame, keyed by class+case. - memex tests: POST api/mesh/run-tests, then polls the ACTIVITY (starts nothing); every request bounded by --timeout; immutable state. - Runner: a case's writer is closed at its verdict, and the class-wide writer admits only the running case's lines (AsyncLocal token), so a case that outlived its bound cannot write onto the next row; a timed-out synchronous listed body is named as still holding its pool slot; verdict words localized beside the glyphs; elapsed formatted invariant. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…rer-only: it executes) Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…live Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…caseOutput/summary) The unkeyed-LogMessage ratchet refused the three verbatim lines. Case lines and the verdict summary are now catalog-keyed (en/de); a failing verdict's own words ride the failure line. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Why
The maintainer's verification page
Admin/Maintenance/refresh-app-tiles-20260828-mainnode-3/Tests(memex-cloud) sat on "Rendering …". A Tests area runs every case and rendered one frame, at the end. So a case that calls an external service live, or one that hangs, looks the same as a stuck page, and the page does not name the case. "Don't we have a proper test runner?"Measured on memex-cloud (pod
memex-portal-deployment-6d84f5d754-xkfk4, 2026-09-26 ~10:52Z), reading the Store/Maintenance cases directly:PaymentPath_Live…: 346 ms (TEST-mode endpoint, verdict Delivering).ModulePins_Live…: 24.3 s, and 17.8 s of it is one read: the portal fetching its own registry's bundle index. Its budget is 20 s. I filed this into triage asrbuergi/Feedback/self-registry-bundle-index-18s-20260926T1108Z.So the page's single frame arrives only after tens of seconds of sequential live work. A second finding: every render re-runs the suite. Two
get …/area/Testsreads 34 s apart each filed a new pair ofAdmin/Maintenance/mnt-…request nodes (10:51:00Z, 10:51:34Z), and the second read answered in 0.9 s with the verdict an EARLIER subscription had cached. A console built by polling the area would therefore re-run live cases on every poll.What
MeshTestRunnerstreams. The first frame shows every case ⏳ pending. After that, one frame per second while a case runs: the running case ▶ with its elapsed time and each output line as it is written, and finished cases ✔/✖. Then one verdict frame (<suite> tests — N/M passed, ✅/❌/⏭ plus a localized word). Rows areControls.DataGrid+PropertyColumnControl<string>. Titles and words are localized and resolved once in the render scope. Elapsed is formatted invariant.AreaFrameClassifier.TestsRunningId(included inIsTransientFrame), and they never contain ✅/❌ or "N/M passed".MeshTestCase+Area(host, suite, cases)is for static-method suites. Synchronous bodies run on the Tests I/O pool; if one times out, the verdict names it as still holding its slot. Each case's writer closes at its verdict. The class-wide[MeshFact]writer admits only the running case's lines (an AsyncLocal token).MeshOperations.RunTests(path)runs the area as an activity. It holds ONE subscription and writes a line per case plus its output to an activity in the caller's partition (categoryTestRun). The terminal status is the verdict (SucceededorFailed), and on timeout the last line names the case still running. It answers{status: Dispatched, activityPath}at once.Mesh.Operationsnow referencesMeshWeaver.GitSyncto reuseActivityRunner.RunActivity.TestsAreaFrame(Layout) is the pure reader of a Tests frame, keyed by class + case.AreaProbetreats a progress frame as transient. A cut-off run names its progress.memex tests @<node>POSTsapi/mesh/run-tests, then polls the ACTIVITY every 2 s and prints each new line. Every request is bounded by--timeout. Exits 0 when all passed, 1 on a failure, 4 when there is no terminal status in time.Architecture/WritingTests.mdhas a new section.DecentralisedTests.mdLane A is updated.Tests (Release
-warnaserror, all clean)MeshTestRunnerTests5/5, includingProgress_streams_pending_running_output_and_one_verdict_last.RunTestsStreamsIntoAnActivityTest1/1: a monolith mesh renders a streaming Tests area, and RunTests turns it into aFailedactivity with the running line (with output), the ✅ and ❌ lines, and1/2 passed.TestsAreaFrameTest2/2.TestsCommandTest2/2.AreaProbeTest7/7.LocalizationTest66/66.Cross-repo
This PR is additive only. Nothing public is removed.
Pairs-with: none — additive only; no public type or member leaves
src/.Mirror-sync: tracked for the Plugins React mirror. Run
npm run sync:i18n -- --ref <merged core sha>in MeshWeaver.Plugins after this merges. The new keys areactivity.tests.{title,case,caseOutput,summary},tests.progress,tests.column.{class,case,result,time,output}andtests.result.{passed,failed,skipped}.Follow-ups in MeshWeaver.Plugins, landing after this is in a sealed set and on the portals:
TestsRunonto this runner. This is the page the maintainer saw.run_tests, which callsMeshOperations.RunTests. It is in the same draft PR. The REST routePOST api/mesh/run-testsis here in core (MeshApiEndpoints, Bearer-only, pinned inMeshApiCookieAuthTest).Addresses to recycle after deploy: none on the core side.
🤖 Generated with Claude Code