Skip to content

[Python] Configure object detection benchmark timeout - #39949

Draft
bvolpato wants to merge 1 commit into
apache:masterfrom
bvolpato:bvolpato/fix-inference-benchmark-timeout
Draft

[Python] Configure object detection benchmark timeout#39949
bvolpato wants to merge 1 commit into
apache:masterfrom
bvolpato:bvolpato/fix-inference-benchmark-timeout

Conversation

@bvolpato

@bvolpato bvolpato commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Addresses #38782.

The object-detection example cancels its Dataflow job after a hard-coded 30-minute wait, even though the benchmark workflow allows 180 minutes per step. Recent benchmark runs reached that cancellation after about 1,803 seconds.

Add --timeout_ms to the example, retaining the 30-minute default, and set the shared object-detection benchmark options to 60 minutes. That override applies to all four CPU/GPU and batch/streaming combinations.

This PR remains draft pending an end-to-end Dataflow run. The fixed-batch configuration fix in #39953 is now upstream and may change runtime. The timeout increase should only merge if the corrected benchmark still needs more than 30 minutes.

The September 2 benchmark includes #39953 and still cancels CPU object detection after 1,804 seconds, but it reports repeated SDK harness disconnections well before that deadline, described in the log as likely process crashes. This does not establish how long a healthy run needs or show that 60 minutes would help. The September 3 run failed during setup and skipped the benchmarks. Worker-failure diagnosis and useful end-to-end timing evidence are still needed before merging.

Validation:

  • Isolated parser and orchestration checks pass for the default, an override, streaming execution, and cleanup after a wait failure. These verify timeout forwarding and cancellation without running a model or Dataflow job.
  • Python compilation, YAPF, Ruff, and git diff --check pass.

End-to-end validation requires the Dataflow benchmark infrastructure.

@bvolpato

bvolpato commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

The failed Python 3.10 wheel job completed the build and auditwheel repair, then timed out during cibuildwheel's Docker cleanup. I did not find a build failure related to this change.

Could someone with rerun access retry that job? My fork account cannot rerun upstream Actions. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant