Skip to content

Spec 066 layer V: transforms validation matrix and behaviour ledger - #1653

Open
Alexandre Zollinger Chohfi (azchohfi) wants to merge 23 commits into
mainfrom
azchohfi-spec-066-layer-v-validation-ledger
Open

Alexandre Zollinger Chohfi (azchohfi) wants to merge 23 commits into
mainfrom
azchohfi-spec-066-layer-v-validation-ledger

Conversation

@azchohfi

@azchohfi Alexandre Zollinger Chohfi (azchohfi) commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Layer V of spec 066 (§12.1, §13 row V). Test-only, no product change. It adds the harness that later layers use to prove they change behaviour only on purpose:

  • A seeded matrix generator. It covers all 378 Transform × Transition × Feature triples, with pairwise coverage of wrapper, flow direction, reduced motion, user code and sub-variant. The control group keeps every other dimension at its default.
  • A WinUI recorder that runs each case through mount, change, remove/add feature, remove modifier, unmount, rent plain and rent transformed. It records threw, pose, opacity (the settled opacity, so a lost update reads 1.00 instead of 0.80), active, motion, curve, running, residue, hit/uia/ttv, channel, pool and diag. Facade Translation drag cases also record screen, the PrintWindow pose (§12.2, Q1f). Motion is controller-driven, layout Offset included, so no verdict depends on wall-clock timing. A stale or drifting capture, a failed screen capture or a compositor that stops rendering makes a case inconclusive, and it reruns, instead of recording a wrong value. Under a facade Translation drag, curve is recorded but not compared, because the drag migrates at a varying moment.
  • An analytic composition model. Each disagreement on main, a throw included, must be attributed by exactly one rule in transforms-ledger.known.json: [Bug] XAML implicit transitions crash with any compositor modifier (LayoutAnimation, Animate, Transition, InteractionStates, Keyframes) #1623, [Bug] UIElement.Translation writes are silently ignored after Reactor enables visual translation #1642, [Bug] A declared .Opacity() change overrides a looping .Keyframes opacity animation #1651, A finished one-shot .Keyframes() opacity entry overrides the declared .Opacity() #1660, §9 row 1 or §9 row 3.
  • Positive and negative controls for every oracle, run before anything is recorded or compared. A session with no compositor frames (locked or disconnected) fails them, so nothing gets recorded there.
  • A pool soak, platform probes (§5.2, §5.3, §6.3) and the behaviour ledger. The ledger was recorded on a hosted runner with main's behaviour. Record mode writes the ledger only when every check passes, and accepts a change only once two observations agree.
  • A Reactor.Tests gate that diffs the PR's ledger against the base commit's. Every change must be declared in transforms-ledger.expected.json with its §9 row or issue. The failure message includes a ready-to-paste entry. Expectations inherited unchanged from the base are ignored, so a merged layer's entries never fail an unrelated PR. The control group must stay byte-identical.
  • A Transforms matrix workflow that runs the full matrix on PRs labelled transforms-matrix and nightly. Label a PR transforms-matrix-record to record a new ledger on a hosted runner and download it as an artifact.
  • A TESTING.md section and §12.1 F in the spec describing what layer V measured. That includes a finding for §12.2: Q1f doesn't reproduce on hosted runners, so the screen oracle is gated on a write-and-reset control and Q1f is logged.

Linked issue / spec

Part of #1630. Implements docs/specs/066-transforms-on-composition-visual.md §12.1 (layer V). Filed while building it: #1651, #1652, #1660 and #1665.

Test plan

  • dotnet test tests/Reactor.Tests --filter-class "*TransformsMatrixTests*": 58 of 58 pass. These cover the generator, the model, the ledger format, the opacity reading, and the declaration gate, including near-miss selectors, inherited expectations, the byte-identical control group, NaN and skip handling, and the drag-case curve exemption.
  • Mutation checks on the gate: removing stale detection, removing control-group protection or loosening the tolerance each turns tests red.
  • Transforms matrix record run 44905273571 on a hosted runner: every instrument control passed (including Screen and SettledOpacity), 378 cases, 0 failures, and every known-difference rule matched. That ledger is the one committed here.
  • Transforms matrix compare run 44955107840 on the head: 378 cases, 0 failures.
  • dotnet build tests/Reactor.AppTests.Host -c Release -p:Platform=x64: clean. I couldn't run the full Reactor.slnx Release build locally because restore can't reach nuget.org (TLS), so the CI Build solution job is the check for that.
  • CI on the head: unit tests, selftests, packaged, AOT, E2E and Build solution all pass.

Risk / breaking changes

None for users. CI cost: the sample (9 cases plus a short soak, about a minute) runs in each selftest shard, including packaged and AOT. The full matrix runs only in its own label-gated or nightly workflow. One new step in the Unit Tests job fetches the base commit's ledger and expectations.

Spec 066 section 12.1: a seeded 378-case matrix of transform modifiers,
implicit transitions and compositor features (every triple, plus pairwise
coverage of wrapper, flow, reduced motion, user code and sub-variant), an
observation recorder, an analytic composition model, positive and negative
controls for every oracle, a pool soak, platform probes, and the behaviour
ledger captured on main. Test-only, no product change.

The normal selftest corpus checks a 9-case sample; the Transforms matrix
workflow runs the full matrix on labelled PRs and nightly. Reactor.Tests
checks that every ledger change from a PR's base is declared.

Part of #1630.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Expectations inherited unchanged from the base are ignored, so a merged
  layer's entries never fail an unrelated PR; CI passes the base copies.
- Known-difference rules must attribute each model disagreement exactly
  once, with {a,b} alternatives; #1623 no longer covers user-code cases,
  and drags lost on a handoff cite #1642.
- Motion is measured from where each animation starts, layout Offset is
  seeked like every channel, and a curve field records the 25% shape.
- New residue field after removeModifier (#1626), no visual reads on
  stagger renters, retries on inconclusive oracles, cheaper record reruns.
- Instrument controls for the curve, pool residue and layout Offset.
- The control group keeps default dims; the baselines are read from the
  checkout; negative selector tests; clearer parse errors.
- Workflow: persist-credentials false, other labels don't cancel a run,
  and a transforms-matrix-record label records on a hosted runner. The
  ledger is re-recorded there, so it's empty in this commit.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@azchohfi Alexandre Zollinger Chohfi (azchohfi) added the transforms-matrix-record Record the spec 066 transforms ledger on a hosted runner and upload it as an artifact label Nov 21, 2026
@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

📦 Build metrics

Artifact sizes for 61c556a vs the base branch (28a8722).

Packages (compressed .nupkg)

Artifact base PR Δ
Microsoft.UI.Reactor.nupkg 2.09 MB 2.09 MB -13 B (0.00%) ≈
Microsoft.UI.Reactor.Advanced.nupkg 528.6 KB 528.6 KB -9 B (0.00%) ≈
Microsoft.UI.Reactor.Devtools.nupkg 287.8 KB 287.8 KB -12 B (0.00%) ≈

Assemblies in Microsoft.UI.Reactor

Artifact base PR Δ
Reactor.Analyzers.dll 437.5 KB 437.5 KB +0 B (0.00%) ≈
Reactor.dll 2.83 MB 2.83 MB +0 B (0.00%) ≈
Reactor.Localization.Generator.dll 16.0 KB 16.0 KB +0 B (0.00%) ≈
Reactor.Wrappers.Abstractions.dll 10.5 KB 10.5 KB +0 B (0.00%) ≈
Reactor.Wrappers.Generator.dll 164.0 KB 164.0 KB +0 B (0.00%) ≈

Assemblies in Microsoft.UI.Reactor.Advanced

Artifact base PR Δ
Reactor.Advanced.dll 1.08 MB 1.08 MB +0 B (0.00%) ≈

Assemblies in Microsoft.UI.Reactor.Devtools

Artifact base PR Δ
Microsoft.UI.Reactor.Devtools.dll 787.0 KB 787.0 KB +0 B (0.00%) ≈

No size change beyond the noise floor. ✅

✅ smaller / ⚠️ larger / ≈ within noise. Sizes come from a Release dotnet pack on the CI runner: packages are the compressed .nupkg download size, assemblies the uncompressed DLL inside it.
workflow run.

@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

🧪 Merged coverage

Coverage for 61c556a vs the base branch (28a8722) — unit + selftest merged.

Metric base PR Δ
Line 88.70% 88.71% +0.01 pp ≈
Branch 45.68% (18380/40239) 45.68% (18381/40239) 0.00 pp ≈

No coverage change beyond the noise floor. ✅

✅ higher / ⚠️ lower / ≈ within noise. Δ is in percentage points; coverage is unit + selftest merged (Debug x64) on the CI runner. Cobertura reports attached to the workflow run as artifacts.

Recorded by the Transforms matrix workflow in record mode (run 44879140336)
on main's behaviour plus this branch's harness: every instrument control
passed, 378 cases, no failures, and every known-difference rule matched.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@azchohfi Alexandre Zollinger Chohfi (azchohfi) added transforms-matrix Run the spec 066 transforms matrix against the ledger (Transforms matrix workflow) and removed transforms-matrix-record Record the spec 066 transforms ledger on a hosted runner and upload it as an artifact labels Nov 21, 2026
@azchohfi
Alexandre Zollinger Chohfi (azchohfi) marked this pull request as ready for review November 21, 2026 07:30
@azchohfi Alexandre Zollinger Chohfi (azchohfi) added the agent-preparing Agent is addressing feedback or completing required validation and CI label Nov 21, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The ledger gate can miss one-DIP changes to protected control fields, while recording can persist unstable samples or observations that failed validation.

3 open issues
What changed in this PR

Adds spec 066 layer V’s validation infrastructure for Reactor’s planned move to composition-visual transforms. It establishes a behavior baseline for later implementation layers without changing product code.

Changes:

  • Adds a seeded 378-case matrix, analytic model, WinUI recorder, instrument controls, pool soak, and platform probes.
  • Introduces ledger comparison and change-declaration checks, with resources shared across test hosts.
  • Adds label-gated and nightly matrix runs, plus documentation of usage, measurements, and remaining coverage limits.
File Description
tests/​Reactor.Tests/​TransformsMatrix/​TransformsMatrixTests.cs Tests matrix generation, modeling, ledger formats, and declaration gates.
tests/​Reactor.Tests/​Reactor.Tests.csproj Links pure-managed matrix code into unit tests.
tests/​Reactor.PackagedTests.Host/​Reactor.PackagedTests.Host.csproj Embeds ledger resources for packaged runs.
tests/​Reactor.AppTests.Host/​SelfTest/​TransformsMatrix/​TransformsLedger.cs Implements ledger serialization, comparisons, and rule matching.
tests/​Reactor.AppTests.Host/​SelfTest/​TransformsMatrix/​MatrixCase.cs Defines dimensions, seeded generation, and sample selection.
tests/​Reactor.AppTests.Host/​SelfTest/​TransformsMatrix/​CompositionModel.cs Models expected geometry and animation precedence.
tests/​Reactor.AppTests.Host/​SelfTest/​SelfTestFixtureRegistry.cs Registers the matrix selftest.
tests/​Reactor.AppTests.Host/​SelfTest/​Fixtures/​TransformsMatrixRecorder.cs Records lifecycle behavior, motion, geometry, pooling, and diagnostics.
tests/​Reactor.AppTests.Host/​SelfTest/​Fixtures/​TransformsMatrixProbes.cs Adds composition probes and optional screen-capture evidence.
tests/​Reactor.AppTests.Host/​SelfTest/​Fixtures/​TransformsMatrixFixture.cs Orchestrates validation, retries, recording, and pool soaks.
tests/​Reactor.AppTests.Host/​SelfTest/​Baselines/​transforms-ledger.known.json Attributes known model disagreements to issues or spec rows.
tests/​Reactor.AppTests.Host/​SelfTest/​Baselines/​transforms-ledger.expected.json Initializes the per-PR change declarations.
tests/​Reactor.AppTests.Host/​Reactor.AppTests.Host.csproj Embeds ledger resources for the selftest host.
TESTING.md Documents matrix modes, recording, and change declarations.
docs/​specs/​066-transforms-on-composition-visual.md Records layer V measurements and remaining validation work.
.github/​workflows/​transforms-matrix.yml Runs full comparisons or recordings and uploads artifacts.
.github/​workflows/​ci.yml Fetches base ledger files for the unit-test declaration gate.

Review effort: Balanced

- Write the ledger only when every check passes; otherwise write transforms-ledger.rejected.json.
- Record mode requires two agreeing observations before a change is recorded.
- Compare the control group byte-for-byte in the gate.
- Epsilon float comparisons, LINQ filters, narrower catch in the pool soak.
- Upload rejected ledgers from the matrix workflow.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The tests can accept stale or screen-inaccurate captures and commit new cases without confirming reproducibility, undermining the reliability of their verdicts.

No open issues 🎉

3 previously missed issues

In code that hasn't changed since last review

Medium severity Require two matching observations before recording new cases

tests/​Reactor.AppTests.Host/​SelfTest/​Fixtures/​TransformsMatrixFixture.cs:162

When a case has no committed record, the expected is null branch stops after one observation, and the confirmation check then accepts it unconditionally. New matrix cases can therefore enter the ledger without demonstrating that their results reproduce. Require two matching observations for new cases; a single observation should suffice only when it matches an existing baseline.

Medium severity Use guarded screen captures for translation drag observations

tests/​Reactor.AppTests.Host/​SelfTest/​Fixtures/​TransformsMatrixRecorder.cs:288

TranslationDrag cases derive their post-write pose and model verdict from RenderTargetBitmap, which can show movement that is absent on screen after translation is enabled (spec 066 §12.2, Q1f). These cases can therefore report model agreement while the visible drag is broken. Use a guarded screen capture for these observations, validate its Q1f control, and skip the screen-dependent checks when no interactive desktop is available.

Medium severity Treat frame timeouts as inconclusive, not successful captures

tests/​Reactor.AppTests.Host/​SelfTest/​Fixtures/​TransformsMatrixRecorder.cs:692

If rendering stops after the instrument controls pass, the two-second timeout makes Frames return as if the requested frames arrived. Later seeks can therefore compare or record stale captures without reporting an instrument failure. Throw on timeout and ensure the recorder and pool soak treat it as an inconclusive observation, not as a product threw value.

3 resolved since last review

Review effort: Balanced

- Facade Translation drag cases record the PrintWindow pose (screen), guarded on an interactive
  desktop, with a Screen instrument whose positive control must reproduce Q1f (spec 066 §12.2).
- A frame timeout throws and makes the observation inconclusive instead of reading stale captures.
- Record mode needs two agreeing observations for a case the ledger doesn't have yet.
- Narrower catch clauses and Path.Join for baseline paths.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@azchohfi Alexandre Zollinger Chohfi (azchohfi) added transforms-matrix-record Record the spec 066 transforms ledger on a hosted runner and upload it as an artifact and removed transforms-matrix Run the spec 066 transforms matrix against the ledger (Transforms matrix workflow) labels Nov 21, 2026
The hosted runner has no interactive desktop, so drag cases record screen=skip (run 44884004263);
every other field is unchanged. A skip now matches any value either way, and recording replaces
a recorded skip with a real pose.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@azchohfi Alexandre Zollinger Chohfi (azchohfi) added transforms-matrix Run the spec 066 transforms matrix against the ledger (Transforms matrix workflow) and removed transforms-matrix-record Record the spec 066 transforms ledger on a hosted runner and upload it as an artifact labels Nov 21, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Active findings can cause false ledger failures from screen-only differences and locale-dependent formatting, while NaN coordinates can incorrectly match finite values and hide real differences.

3 open issues

Review effort: Balanced

@azchohfi Alexandre Zollinger Chohfi (azchohfi) added the transforms-matrix-record Record the spec 066 transforms ledger on a hosted runner and upload it as an artifact label Nov 21, 2026
…reeing recordings

- Known rule for #1660: a one-shot keyframe entry added while .Opacity() declares 0.8 settles at
  its last value, 1 (record run 44901927811). Every other opacity disagreement follows a step that
  threw, and an existing rule already covers it.
- Record mode accepts a change once any two conclusive observations agree, so one outlier between
  two agreeing runs doesn't reject it (ScaleRotationCenter.Opacity.Animate in that run).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Hosted record run 44905273571: every rest step gains opacity, and model gains its opacity part
where they disagree. All of those follow a step that threw, except the #1660 keyframe cases. The
soak counts are unchanged. No other field changed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@azchohfi Alexandre Zollinger Chohfi (azchohfi) added transforms-matrix Run the spec 066 transforms matrix against the ledger (Transforms matrix workflow) and removed transforms-matrix-record Record the spec 066 transforms ledger on a hosted runner and upload it as an artifact labels Nov 21, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Pending Windows JIT, packaged, and NativeAOT validation leaves the state-mutating compositor harness’s baseline unconfirmed, while live WinUI verification and independent cross-family review remain unavailable.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

@azchohfi Alexandre Zollinger Chohfi (azchohfi) added ready-for-review Agent work and technical checks complete; awaiting review, not approval or merge and removed agent-preparing Agent is addressing feedback or completing required validation and CI labels Nov 21, 2026
The handoff migrates the drag at a moment that varies between runs, so the curve is measured from a
different start and flips (layer 2: Translation.Rotation.Animate 0.2 or 0.0,
Translation.Translation.Stagger 0.4 or 0.0). Comparisons skip curve for user=TranslationDrag cases
and recording keeps the committed value. The motion verdict is still compared, and the ledger is
unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@azchohfi Alexandre Zollinger Chohfi (azchohfi) added agent-preparing Agent is addressing feedback or completing required validation and CI and removed ready-for-review Agent work and technical checks complete; awaiting review, not approval or merge labels Nov 21, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because exhausted stabilization attempts can leave drifting measurements unflagged, allowing unreliable comparisons or recordings to be accepted.

No open issues 🎉

Review effort: Balanced

If two captures never agree within five tries, the sample is drifting, so the case is now
inconclusive and reruns instead of using the last reading. The ledger is unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because failed screen captures can be recorded as valid empty results, potentially overwriting valid baselines or causing false regression failures.

No open issues 🎉

Review effort: Balanced

PrintWindow failing now throws, so a drag case's screen observation becomes inconclusive and
reruns instead of recording screen=none. The instrument guard and the probes treat it as no
screen. The ledger is unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because catch-all baseline rules conflate keyed-Memo exit-animation failures with separate defects, obscuring whether the attributed bugs are actually resolved.

No open issues 🎉

Review effort: Balanced

@azchohfi Alexandre Zollinger Chohfi (azchohfi) added ready-for-review Agent work and technical checks complete; awaiting review, not approval or merge and removed agent-preparing Agent is addressing feedback or completing required validation and CI labels Nov 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-review Agent work and technical checks complete; awaiting review, not approval or merge transforms-matrix Run the spec 066 transforms matrix against the ledger (Transforms matrix workflow)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants