Skip to content

Analyze delegated coverage once per test run - #1891

Draft
wenytang-ms wants to merge 2 commits into
mainfrom
wenyt/delegate-test-p0
Draft

Analyze delegated coverage once per test run#1891
wenytang-ms wants to merge 2 commits into
mainfrom
wenyt/delegate-test-p0

Conversation

@wenytang-ms

@wenytang-ms wenytang-ms commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Why

Coverage for a delegated test run was collected and published from inside the loop that runs each batch of tests, one batch per project and per set of classes. A run split over several batches therefore published a report per batch. Because file coverage is stored rather than merged, the last batch hid everything the batches before it had covered.

The pre-run cleanup lived in the same loop, which made it worse in two ways: an explicit appendResult: false threw away the data the run itself had just produced, and because the directory was fixed per project rather than per run, two runs started close together could erase each other.

Analyzing no execution data at all reported every line as uncovered — indistinguishable from a run that genuinely covered nothing. A run whose agent never attached looked like a real 0%.

What

One directory per run, analyzed once. A delegated run gets a directory of its own, and every project's data is analyzed after all batches have finished rather than after each one.

appendResult becomes a matter of scope, not deletion. Analyze only this run's directory, or the root that still holds the runs before it. Nothing is deleted while a run might be writing into it, which is what let concurrent runs erase each other. Old run directories are collected on a time bound instead, and a directory younger than that bound is never touched.

No execution data is an error. A run that produced nothing reports the failure rather than a fabricated 0%.

The contract

Delegated coverage now goes through an explicit request rather than a bare output path:

interface CoverageRequest {
    format: 'jacoco-exec';
    version: 1;
    executionDataDirectory: vscode.Uri;
    agentJar: vscode.Uri;
}
  • The agent jar comes from here, so the agent that writes the data and the analyzer that reads it are always the same version. A runner does not choose, resolve or configure one.
  • This extension owns the directory — it creates it, decides how widely to analyze, and cleans up. A runner writes .exec files into it and must not clear it or write with append=false.
  • Nothing about reporting is in the payload: no XML, no source directories, no report tasks. The runner produces execution data; analysis happens here.
  • Cancellation is IRunTestContext.cancellationToken, not part of the coverage request.

Runners advertise support with the jacoco-exec@1 capability, so the version is negotiated rather than assumed.

Testing

  • tsc and eslint clean.
  • mvnw clean verify builds the Java plugin clean, including checkstyle.
  • New CoverageHandlerTest case covers the no-execution-data failure.

wenytang-ms and others added 2 commits July 27, 2026 13:57
A delegated runner previously received a bare output directory and had to
supply its own JaCoCo agent. That left the agent free to be a different
version than the analyzer, and left one directory shared between every run.

Hand it a versioned artifact request instead. It names the artifact, a
directory that belongs to a single test run, and the agent this extension
ships, which is the only one guaranteed to match the analyzer. Creation,
retention and removal of the directory stay on this side, so a runner never
deletes data another run may still be writing.

The capability becomes `jacoco-exec@1`, pinning the revision of the contract
rather than the feature, so a later incompatible revision can be advertised
under a name of its own.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8b7a64f2-3b9f-4328-a173-f4728a1fa95c
Coverage was collected and published inside the loop that runs each batch of
tests. A run split over several batches therefore published a report per
batch, and because file coverage is stored rather than merged, the last batch
hid what the batches before it had covered. The pre-run cleanup sat in the
same loop, so an explicit `appendResult: false` also threw away the data this
very run had just produced.

Collect once per project and analyze after every batch has finished. For a
delegated run, `appendResult` becomes a matter of how wide to analyze -- this
run's directory, or the root that still holds the runs before it -- rather
than of deleting data, which is what made concurrent runs able to erase each
other.

Analyzing no execution data at all reported every line as uncovered, which
reads exactly like a run that genuinely covered nothing. Report it as the
failure it is, so a run whose agent never attached cannot be mistaken for 0%.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 8b7a64f2-3b9f-4328-a173-f4728a1fa95c
@wenytang-ms
wenytang-ms force-pushed the wenyt/delegate-test-p0 branch from ec906c1 to d0e7e25 Compare July 27, 2026 05:57
@wenytang-ms wenytang-ms changed the title Analyze delegate-test coverage from merged .exec files Analyze delegated coverage once per test run Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant