Skip to content

fix(build): propagate DUT worker failures - #198

Open
FU-max-boop wants to merge 1 commit into
XS-MLVP:mainfrom
FU-max-boop:fix/dut-build-exit-status
Open

FU-max-boop wants to merge 1 commit into
XS-MLVP:mainfrom
FU-max-boop:fix/dut-build-exit-status

Conversation

@FU-max-boop

Copy link
Copy Markdown

Summary

  • retain every multiprocessing.AsyncResult and inspect it after all DUT workers finish
  • fail the parent process when any supported DUT build fails
  • distinguish six intentionally unimplemented build scripts with UnsupportedDUTError, preserving wildcard and subtree builds
  • add hardware-independent regression coverage for worker outcomes, result transport failures, deterministic aggregation, unsupported builders, and prebuilt DUTs

Problem

build_dut() submitted _build_dut() through Pool.apply_async() but discarded every returned result. _build_dut() also only logged falsey returns and exceptions. A supported DUT build could therefore fail while the parent returned normally and the shell reported status 0.

A process-level probe using a real Linux multiprocessing.Pool and a synthetic supported builder returning False reproduced the behavior:

e887def (base): logs the build failure, shell exit 0
31aaffb (head): raises "Failed to build DUTs: build_ut_audit", shell exit 1

Implementation

  • _build_dut() returns True for success, False for a falsey result or ordinary build exception, and None only for an explicit UnsupportedDUTError.
  • build_dut() records results under deterministically sorted DUT names, waits for all workers, then calls AsyncResult.get() for every submitted build.
  • Supported failures are aggregated into one RuntimeError; exceptions from AsyncResult.get() propagate.
  • The six existing placeholder scripts explicitly raise UnsupportedDUTError, so broad targets continue to skip them.

Validation

python -m pytest -p no:cacheprovider -q tests/test_build_dut.py
  • Python 3.10.20 / pytest 8.3.4: 16 passed
  • Python 3.12.3 / pytest 9.1.1: 16 passed
  • final regression file applied to base e887def: 14 failed, 2 passed
  • real multiprocessing process probe: shell exit 0 -> 1
  • real comm re-export plus all six placeholder imports: passed
  • git diff --check: passed

Related work and merge ordering

#196 propagates pytest exit codes later in run.py; this change covers the earlier DUT-build stage and does not overlap its files.

A follow-up RAS contribution replaces scripts/build_ut_frontend_bpu_ras.py with a real builder. This PR should land first; the RAS branch will then retain its builder and drop that script from the unsupported set during rebase.

Scope boundary

  • Targeted tests do not compile RTL or invoke Picker.
  • The process-level probe uses a real multiprocessing pool with a synthetic failing build function, not a full hardware compilation.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant