Skip to content

[Python] Propagate WriteToFiles finalization failures - #39993

Draft
bvolpato wants to merge 2 commits into
apache:masterfrom
bvolpato:bvolpato/python-fileio-finalization-errors
Draft

[Python] Propagate WriteToFiles finalization failures#39993
bvolpato wants to merge 2 commits into
apache:masterfrom
bvolpato:bvolpato/python-fileio-finalization-errors

Conversation

@bvolpato

@bvolpato bvolpato commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

WriteToFiles currently catches every BeamIOError from final renames and emits successful FileResults even when the destination files are absent. A genuine I/O failure can therefore leave data in temporary files while the pipeline reports success.

Propagate rename failures so the bundle can retry. Before renaming, skip a file only when its temporary source is absent and its final destination exists, preserving retries after complete or partial finalization. An existing destination does not suppress errors while the source remains.

Five regression tests cover failed renames without successful output, already-completed retries, partial moves followed by retry, stale destinations, and missing files. Against the unfixed implementation, four fail and the completed-retry compatibility test passes.

Testing

From sdks/python, with the SDK and test dependencies installed:

python -m pytest apache_beam/io/fileio_test.py -q

Python 3.12: 49 passed, 2 skipped. YAPF 0.43.0, Ruff 0.15.22, and git diff --check pass for the changed files.

Downsides

Finalization adds one existence lookup per temporary file, plus a destination lookup when that source is absent. Remote filesystems incur these metadata requests before the existing batched rename. Genuine finalization errors now fail the bundle instead of being suppressed.


  • Describe the bug and include reproducible regression tests; no separate Python issue is linked.
  • Update CHANGES.md with the behavior change.
  • Apache Individual Contributor License Agreement, if required for this contribution.

See the Contributor Guide and CI documentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant