Skip to content

Spurious timeout in the asmjs builder #48826

Closed
Closed
Spurious timeout in the asmjs builder#48826
Labels
A-spuriousArea: Spurious failures in builds (spuriously == for no apparent reason)O-asmjsTarget: asm.js - http://asmjs.org/O-emscriptenTarget: 50% off wasm32-unknown-musl. the savings come out of stdio.h, but hey, you get SDL!T-infraRelevant to the infrastructure team, which will review and decide on the PR/issue.

Description

@alexcrichton
alexcrichton
opened on Mar 7, 2018

The asm.js builder has been creeping up in time it takes to run ever since it was originally added, and clocking in at around 2.5 hrs now this builder is ripe for timing out on occasion. The suspected cause here is the fact that the run-pass test suite takes >1hr to run and the suspected cause there is that we're effectively compiling every test with LTO which can take quite some time.

This issue is intended to track possible solutions to bring the builder back under control. Today in a discussion a few possible avenues arose:

  • Shard the asm.js builder into two builders, each running half the run-pass tests
  • Stop running the full test suite for asm.js altogether, we seem to regress this very rarely
  • Stop optimizing tests, debug mode may be much faster.

cc @kennytm

Activity

  1. alexcrichton
    added
    A-spuriousArea: Spurious failures in builds (spuriously == for no apparent reason)
    O-asmjsTarget: asm.js - http://asmjs.org/
    T-infraRelevant to the infrastructure team, which will review and decide on the PR/issue.
    on Mar 7, 2018

  2. kennytm commented on Mar 7, 2018

    @kennytm
    kennytm
    on Mar 7, 2018
    Member

    For reference, the rough timing breakdown according to the log in the recent successful builds are:

    Details

    (Unit = seconds)

      69261 69207 69163 69145
    make-prepare 51.80 51.39 44.70 51.33
    stage0-test 11.46 11.49 9.95 11.95
    stage0-rustc 837.85 894.54 736.97 891.64
    llvm 334.30 331.78 299.45 320.13
    stage0-trans-llvm 61.98 66.88 56.99 65.78
    llvm-emscripten 155.00 154.79 148.31 157.17
    stage0-trans-emscripten 55.33 56.83 49.81 57.00
    stage1-std 91.74 90.80 74.97 88.20
    stage1-test 17.12 17.73 14.51 17.77
    stage1-rustc 1306.78 1379.93 1115.28 1350.69
    stage1-trans-llvm 93.22 99.94 81.47 96.71
    stage1-trans-emscripten 81.14 86.71 73.54 84.38
    stage2-std 58.86 57.58 47.54 56.76
    stage2-test 12.49 12.86 10.79 12.88
    stage0-compiletest 45.64 47.33 39.31 46.43
    test/ui 237.82 219.62 197.82 218.42
    test/run-pass 3697.79 3738.31 3276.24 3717.92
    test/compile-fail 114.26 110.62 102.75 110.47
    test/parse-fail 3.86 3.88 3.55 3.80
    test/run-fail 167.54 172.85 148.10 170.07
    test/run-pass-valgrind 14.39 14.91 12.64 14.69
    test/mir-opt 67.39 69.09 59.82 69.13
    test/codegen 3.23 3.24 2.88 3.23
    test/codegen-units 18.60 19.27 16.86 19.12
    test/incremental 119.36 121.20 109.48 121.45
    test/debuginfo 153.45 167.77 135.39 162.18

    We see that finishing the test/run-pass tests take roughly 1 hour.

    We could compare with those of

    android... 
    make-prepare                	  54.11
stage0-test                 	  12.21
stage0-rustc                	 772.94
llvm                        	 295.78
stage0-trans-llvm           	  56.48
stage1-std                  	  81.96
stage1-test                 	  14.97
stage1-rustc                	1152.85
stage1-trans-llvm           	  87.40
stage2-std                  	  63.00
stage2-test                 	  14.74
stage2-remote-test-server   	   2.61
stage0-remote-test-client   	   2.42
stage0-compiletest          	  45.28
test/ui                     	 101.67
test/run-pass               	 660.46
test/compile-fail           	 135.16
test/parse-fail             	   5.25
test/run-fail               	  19.38
test/run-pass-valgrind      	   3.89
test/mir-opt                	  20.29
test/codegen                	   3.49
test/codegen-units          	   2.16
test/incremental            	  17.71
test/debuginfo              	 111.61
    x86_64... 
    make-prepare                	  52.89
stage0-tidy                 	  54.54
bootstrap                   	  11.89
stage0-rustc                	 883.23
llvm                        	 333.87
stage0-trans-llvm           	  64.13
stage1-std                  	 145.13
stage1-test                 	  17.89
stage1-rustc                	1369.94
stage1-trans-llvm           	  97.95
stage0-compiletest          	  47.20
test/ui                     	 101.08
test/run-pass               	 614.08
test/compile-fail           	 135.41
test/parse-fail             	   5.07
test/run-fail               	  14.34
test/run-pass-valgrind      	   3.50
test/mir-opt                	  20.33
test/codegen                	   4.22
test/codegen-units          	   2.40
test/incremental            	  14.98
test/debuginfo              	  13.54
test/ui-fulldeps            	  39.77
test/run-pass-fulldeps      	 330.72
test/run-fail-fulldeps      	   4.92
test/compile-fail-fulldeps  	  86.63
test/incremental-fulldeps   	   3.24
stage2-rustdoc              	 160.96
test/rustdoc                	 125.74
test/run-make               	 311.71
...
    wasm32... 
    make-prepare                	  50.83
stage0-test                 	  11.30
stage0-rustc                	 841.60
llvm                        	 307.38
stage0-trans-llvm           	  61.75
lld                      	  19.76
stage1-std                  	  88.76
stage1-test                 	  16.50
stage1-rustc                	1263.30
stage1-trans-llvm           	  89.47
stage2-std                  	  54.54
stage2-test                 	  13.24
stage0-compiletest          	  45.60
test/ui                     	  90.01
test/run-pass               	 865.87
test/compile-fail           	 122.83
test/parse-fail             	   4.91
test/mir-opt                	  19.47
test/codegen-units          	   1.97
stage2-rustdoc              	 150.73
stage2-test-core            	  82.39
...

    and we see that the run time of the run-pass suite typically take 10 to 15 minutes to complete, so the running time of asmjs's run-pass is 5 times longer than other builders.

  3. alexcrichton commented on Mar 7, 2018

    @alexcrichton
    alexcrichton
    on Mar 7, 2018
    MemberAuthor

    FWIW pre-LLD the wasm32-unknown run-pass test suite took 2700s, wheras after LLD it took 850s, a transition from 45m to ~15m. That's I think definitely confirming that Emscripten is so slow because of how it's linker works, it's just LLVM's LTO linker.

    Once my machine frees up from timing other things I'll run some tests to see how much optimizing tests costs us here.

  4. alexcrichton commented on Mar 8, 2018

    @alexcrichton
    alexcrichton
    on Mar 8, 2018
    MemberAuthor

    Disabling optimizations in tests unfortunately broke a bunch of tests

    failures:
    [run-pass] run-pass/deep.rs
    [run-pass] run-pass/intrinsics-integer.rs
    [run-pass] run-pass/intrinsics-math.rs
    [run-pass] run-pass/issue-32947.rs
    [run-pass] run-pass/issue-38074.rs
    [run-pass] run-pass/issue-39720.rs
    [run-pass] run-pass/mir_heavy_promoted.rs
    [run-pass] run-pass/next-power-of-two-overflow-debug.rs
    [run-pass] run-pass/next-power-of-two-overflow-ndebug.rs
    [run-pass] run-pass/packed-struct-borrow-element.rs
    [run-pass] run-pass/simd-intrinsic-generic-arithmetic.rs
    [run-pass] run-pass/simd-intrinsic-generic-comparison.rs
    [run-pass] run-pass/simd-intrinsic-generic-elements.rs

    and shaved 5 minutes off the total build time locally. I'm not sure if that'll actually translate to wins upstream...

  5. alexcrichton
    added a commit that references this issue on Mar 16, 2018

    ci: Disable optimized tests for asm.js

    1c4b9c1
  6. bors
    added a commit that references this issue on Mar 19, 2018

    Auto merge of #49095 - alexcrichton:debug-asmjs, r=kennytm

    c2f4744
  7. bors
    closed this as completedin 1c4b9c1on Mar 19, 2018

  8. alexcrichton commented on Mar 19, 2018

    @alexcrichton
    alexcrichton
    on Mar 19, 2018
    MemberAuthor

    Looks like with #49095 our builds have stabilized at from 2.5 hours to 2 hours on the asmjs builder. That should hopefully buy us at least a few months!

  9. alexcrichton
    added a commit that references this issue on Mar 26, 2018

    rustbuild: Disable docs on cross-compiled builds

    bd28641
  10. kennytm
    added a commit that references this issue on Mar 26, 2018

    Rollup merge of rust-lang#49187 - alexcrichton:no-cross-docs, r=kennytm

    ede4cd9
  11. TimNN
    added a commit that references this issue on Mar 26, 2018

    Rollup merge of rust-lang#49187 - alexcrichton:no-cross-docs, r=kennytm

    bd3db2b
  12. workingjubilee
    added
    O-emscriptenTarget: 50% off wasm32-unknown-musl. the savings come out of stdio.h, but hey, you get SDL!
    on Oct 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-spuriousArea: Spurious failures in builds (spuriously == for no apparent reason)O-asmjsTarget: asm.js - http://asmjs.org/O-emscriptenTarget: 50% off wasm32-unknown-musl. the savings come out of stdio.h, but hey, you get SDL!T-infraRelevant to the infrastructure team, which will review and decide on the PR/issue.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions