Skip to content

Fix and generalize streamed uploads for VirtualActionInput - #31207

Open
fmeum wants to merge 3 commits into
bazelbuild:masterfrom
fmeum:deterministic-writer-pipe
Open

fmeum wants to merge 3 commits into
bazelbuild:masterfrom
fmeum:deterministic-writer-pipe

Conversation

@fmeum

@fmeum fmeum commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Description

RemoteExecutionCache only supported uploading a VirtualActionInput without fully materializing it, using a bounded pipe fed from a virtual thread. This moves the pipe into DeterministicWriter#getInputStream() so that any writer can be streamed the same way.

The shared pipe also reports a failure in the writer to the reader as an IOException. Previously the failure was swallowed and the reader saw a clean EOF, which could result in a truncated blob being uploaded. Closing the stream before the end now interrupts a writer that is blocked on a full buffer.

Motivation

Work towards implementing --file_write_strategy=remote for Bazel.

Build API Changes

No

Release Notes

RELNOTES: None

@fmeum
fmeum force-pushed the deterministic-writer-pipe branch 2 times, most recently from cd70017 to 4b17bd2 Compare September 18, 2026 16:54
@fmeum
fmeum marked this pull request as ready for review September 18, 2026 16:58
@fmeum
fmeum requested a review from a team as a code owner September 18, 2026 16:58
@fmeum
fmeum requested review from tjgq and removed request for a team September 18, 2026 16:59
@github-actions github-actions Bot added team-Remote-Exec Issues and PRs for the Execution (Remote) team awaiting-review PR is awaiting review from an assigned reviewer labels Sep 18, 2026
rules_java 9.9.0 ships java_tools v21.0, whose JavaBuilder bundles
Error Prone 2.50.0 instead of 2.46.0. Error Prone versions before
2.48.0 crash on flexible constructor bodies (JEP 513, final in JDK 25):
MustBeClosedChecker throws a ClassCastException for any AutoCloseable
subclass whose constructor has statements before super(...). This
affects both Bazel's own code and users compiling Java with the default
toolchain, so MODULE.tools is updated as well.
RemoteExecutionCache only knew how to upload a VirtualActionInput
without materializing it, using a bounded pipe fed from a virtual
thread. Move the pipe into DeterministicWriter#getInputStream so that
any writer can be streamed the same way and rename the uploader method
to uploadDeterministicWriter.

The shared pipe also reports a failure in the writer to the reader as
an IOException. Previously the failure was swallowed and the reader saw
a clean EOF, so a truncated blob was uploaded and rejected by the
server. Closing the stream before the end now interrupts a writer that
is blocked on a full buffer.
@fmeum
fmeum force-pushed the deterministic-writer-pipe branch from 4b17bd2 to 5fde1b5 Compare September 18, 2026 19:26
@fmeum fmeum changed the title Stream any DeterministicWriter through the bounded upload pipe Fix and generalize streamed uploads for VirtualActionInput Sep 20, 2026
@fmeum

fmeum commented Sep 20, 2026

Copy link
Copy Markdown
Collaborator Author

Stacked on #31208 as it uses a flexible constructor body.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting-review PR is awaiting review from an assigned reviewer team-Remote-Exec Issues and PRs for the Execution (Remote) team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant