Repository navigation
Test suite overstates coverage: 73/195 functions never invoked, and the coverage ratchet is regex-based #152
Description
Activity
- added a commit that references this issue
on Aug 3, 2026 Partially fixed in 0.74.0 (#154) — leaving open for the remaining coverage gap.
Fixed — the ratchet no longer lies:
Test-MsixInvokednow uses ASTCommandAstdetection instead of a regex over test source text, soGet-Command Add-MsixFoo -Module MSIX, comments, andContext/Ittitles no longer count as coverage.- That immediately exposed the 8 falsely-certified mutators named in this issue. Six gained real behavioural tests (
MSIX.UncoveredMutators.Tests.ps1);Update-MsixMgrandUpdate-MsixSdkToolmoved toPermanentlyExcludedas the same network-updater class already excluded. - Writing those six tests found three real bugs in cmdlets that could never produce a packable package — see Correctness: discarded return values, missing idempotency guards, and @($null) null-deref cluster #153.
Still open:
- 73 of 195 exported functions are still never invoked by a test (33 of them high-risk: mutators, downloaders, signing, disk-image mounting). The six fixed here were the falsely-certified subset only.
- No code-coverage measurement in CI.
- The critical-path gaps listed in this issue remain unaddressed except where 0.74.0 touched them: signing-failure rollback and
-WhatIfbyte-identity now have real tests (added with CRITICAL: three paths silently destroy or corrupt the operator's package #145), but Authenticode rejection is still only exercised with mocks hard-coded toStatus = 'Valid'.
Worth noting the hit rate as justification for continuing: six never-tested mutators yielded three real bugs.
Completed in #155.
- Never-invoked exported functions: 73 → 26. All 26 that remain are genuinely environment-bound — network installers, Hyper-V/VHDX, the ProcMon kernel driver, Windows Sandbox, or a live installed package — and each is listed with its reason rather than left silent.
- The ratchet now guards the whole exported surface, not just
Add/Remove/Set/Updatemutators. That narrow scope is exactly how 73 functions came to be uncovered with nothing reporting it. Three orchestrators (Invoke-MsixAccelerator,Invoke-MsixAutoFixLoop,Invoke-MsixRemediationPlan) are recorded as real debt with a ceiling so the list can only shrink. - Authenticode rejection is tested for the first time. Both existing tests mocked
Get-AuthenticodeSignatureto returnValid, so thethrowbranch — the control that stops a planted toolchain binary from being executed — had never executed. Every rejection status is now driven, plus a valid-signature-but-untrusted-publisher case and a positive control so the suite cannot pass by simply throwing at everything. - Read-only scanner matrix: all 15 scanners against a well-formed package, against the degenerate manifest shape the module generates itself (no
<Applications>/<Extensions>/<Capabilities>— the shape behind the null-deref class), and against a missing package to prove they fail loudly instead of returning an empty result that reads as "clean". Explicit named invocations back the matrix, because& $Namedispatch is invisible to AST detection and teaching the ratchet to trust strings again is the exact weakness this issue was about. - Code coverage is now measured in CI and reported in the job summary. There was no measurement at all before.
- Test isolation fixed: the update-check tests wrote to the real user cache at
%LOCALAPPDATA%\MSIX\update-check.json, poisoning it with a mocked version.
Suite: 765 → 848 passing.
- added a commit that references this issue
on Aug 4, 2026
Measured by AST-parsing
FunctionsToExportagainstCommandAstinvocations in the test tree.Should -Not -ThrowThe coverage ratchet certifies functions on the strength of
Get-CommandandIttitlesMSIX.Tests/MSIX.Contract.CoverageMap.Tests.ps1:59defines "invoked" as a text regex:Get-Command Add-MsixFoo -Module MSIXmatches. So do comments andContext/Ittitles. Eight mutators are falsely certified as covered (regex says yes, AST says never invoked):Set-MsixInstalledLocationVirtualization,Add-MsixLoaderSearchPathOverride,Remove-MsixUpdaterArtifact,Update-MsixMgr,Update-MsixSdkTool,Add-MsixFirewallRule,Add-MsixStartupTask,Update-MsixPackageVersion. The$KnownUncoveredratchet reads empty ("burned down in #102") while 12 mutators are genuinely uninvoked.Critical paths with no real assertion
UnsignedOutputPathreferences areParameters.ContainsKeyor source regex. No test ever fails a signing operation and asserts the original bytes survive.-WhatIfside-effect freedom — no test runs a mutator with-WhatIfand asserts the file is byte-identical. (This is why the broken-WhatIfwent unnoticed.)New-MsixWorkspaceis only ever mocked; nothing asserts cleanup after failure._MsixVerifyAuthenticodeis invoked twice but both mocks hard-codeStatus = 'Valid', so thethrowbranch atMSIX.PsfBinaries.ps1:151never executes.MSIX.WhatIf.Tests.ps1:85-162verifies the atomic pack-sign-move invariant by grepping module source for aMove-Itemline — it passes if the line exists in a dead branch or after an earlyreturn.Other
MSIX.Tests/MSIX.OfflineRegistry.Tests.ps1:173has no assertion at all in the body — it passes unconditionally.MSIX.AuthenticodeFilter.Tests.ps1:47-56reimplements the production filter expression in the test body and asserts the copy — it passes even if the real function is deleted.MSIX.MutatorIntegration.Tests.ps1:45depends on execution order acrossItblocks.Suggested fixes (prioritised)
CommandAstdetection; re-baseline$KnownUncoveredto the true list.-WhatIfbyte-identity test across allSupportsShouldProcessmutators.NotSigned/HashMismatch/UnknownError→ throws).<Properties>/<Extensions>/<Applications>.