Repository navigation
Implement some way to run UI tests ignoring run-pass tests #54047
Description
Activity
- (1a) One option: make a separate
ui-run-passsuite and move this stuff there. That is perhaps the simplest to implement, but I don't know if it has the ideal developer UX.- (1b) A variant of this: reorganize
ui/to have subdirectoriesui/compile-fail/,ui/compile-pass/,ui/run-pass/, ... and then move everything there. Then a developer should be able to opt into the subarea they are interested in, likecompile-fail, by specifying it on their command line.
- (1b) A variant of this: reorganize
- (2a) Another (orthogonal) option: add a flag that makes
compiletesttreat// run-passas if it said// compile-pass. This would resolve the time overhead from running the tests.- (2b) A variant of this would provide flags to let the user say things like "only run the
// compile-failtests" or "skip the// run-passtests...
- (2b) A variant of this would provide flags to let the user say things like "only run the
Of the above choices, I think I like (1b) the best...
Anyway, my apologies: My whole strategy of resolving #53764 by migrating the
run-passtests to a subdirectory ofui/was based on the presumption that theuitests don't represent a huge part of our autobuild infrastructure overhead. While I did ask the infrastructure team whether this presumption was correct, the whole problem is that that presumption was not sufficient; it considered solely the overhead from our autobuild infrastructure, and failed to consider overall developer UX.- (1a) One option: make a separate
Nominating for discussion at T-compiler meeting.
(actually, after further reflection, I'm starting to warm up to option (1a). After all, we started with a model where we had separate compile-fail and run-pass suites. So it would make sense to retain that division, and just think of this as "upgrading" each of the suites to test the compiler UX as well...)
I remember that @nikomatsakis wanted to avoid variants like
1a/bto introduce a per-feature directory structure instead:/ui /rfc-XXXX-feature-name1 *both compile-fail and run-pass tests for this feature* /rfc-YYYY-feature-name1 *both compile-fail and run-pass tests for this feature* /rfc-ZZZZ-feature-name1 *both compile-fail and run-pass tests for this feature*with run-pass/compile-pass/compile-fail discerned by in-file annotations (i.e. like it works right now basically).
With this scheme compiletest/rustbuild would need to regain the ability to run test groups separately, but now using those annotations.I do notice that
compiletesthas a--modeargument which is implemented as only accepting one of "(compile-fail|parse-fail|run-fail|run-pass|run-pass-valgrind|pretty|debug-info|incremental|mir-opt)",I suspect that is implemented by inspecting directory names, but since it does not list
ui, we could expand its meaning to also mean "inspect the header of theui/tests to determine its catgory." That might give us a straight-forward way to implement option (2b) that I recently added to my comment above.Update: Oops, the documentation doesn't mention
ui, but the implementation does have aMode::Uivariant, and theimpl FromStr for Modewill indeed translate"ui"toMode::Ui.Still, I am warming up to this new approach to implementing option (2b).
My feeling here:
- I dislike dividing up our test suite into directories like run-pass, compile-fail, etc
- I would rather have the "primary organization" of the tests be the area of code that they are testing (e.g., borrowck, regionck, some RFC, etc)
This leads me to suggest that we ought to add a "test filter" that lets you pare down the tests to exclude run-pass or other modes. In fact, if we included the "mode" in the name of the test, that would happen automatically via
--test-args, but that might break other things.e.g., the name of a
// run-passtest calledfoo/bar.rscould befoo/bar.rs (run-pass)or something like that. Then you could do--test-args '(run-pass)'.(Of course, there really is no ideal way to organize the tests. But organizing by "what the tests are testing" has advantages when you're trying to look over the set of tests for a given feature, and using
--test-argsto filter seems to fulfill the other use cases. I can't think of a time when I'm like "I'd like to browse the set of tests that don't generate errors".)discussed at meeting. P-high. I will work on addressing @petrochenkov 's developer UX concern in short term.
namely, my short term plan is:
- Turn
src/test/run-pass/into "anotherui-suite. Any test that cannot deal with such a transition should be moved elsewhere.- This is PR
uiify run-pass #54223. (I ended up adding workarounds so that all tests could stay in that directory.)
- This is PR
- Move everything I moved to
src/test/ui/run-pass/back to the newui-ifiedsrc/test/run-pass/.
That the short-term plan.
The long term plan will involve engaging the broader compile team to figure out how we want to organize all of our tests.
- Turn
3 remaining items
@RalfJung yes, one of my assumptions is that as part of a broader reworking, we'll set things up so that one can opt into skipping e.g. all tests that have
// run-passin their header (or as one of their properties; however that knowledge is encoded...)visited for triage. The plan is still to move
src/test/ui/run-pass/back tosrc/test/run-pass/once PR #54223 lands.However, given that landing PR #54223 hasn't been a seamless process: If #54223 hasn't landed in a week's time, then I will immediately move
src/test/ui/run-pass/back tosrc/test/run-passat that point, in order to stop inconveniencing the developer UX for people focused on compile-fail issues in theuisuite.Reacted by Esteban Kuber
run-passtest were recently merged into UI tests (#53860, #53992, #53994), which is unfortunate in several aspects.If you are working with diagnostics (changing spans, labels, error messages, etc) or in "compile-fail" area in general, and want to check the result on UI tests, then there's absolutely no need to run thousands of run-pass tests (+ their NLL variations) as well, which are also quite slow because you have to launch both the compiler and the produced executable.