Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix using GcsIO from a DoFnin Python #32362

Merged
merged 4 commits into from
Aug 30, 2024
Merged

Conversation

iht
Copy link
Contributor

@iht iht commented Aug 29, 2024

GcsIO is failing when trying to use from a DoFn in Dataflow, due to repeated parsing of the pipeline options. This change avoids parsing the options just to retrieve an already-parsed option.

This change fixes #32361.

@iht
Copy link
Contributor Author

iht commented Aug 29, 2024

+R: @liferoad

Copy link
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

@iht
Copy link
Contributor Author

iht commented Aug 29, 2024

The GcsIO tests are failing, I will add another commit fixing them.

Copy link
Collaborator

@liferoad liferoad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fix the pre-commit. LGTM

@iht
Copy link
Contributor Author

iht commented Aug 30, 2024

All tests are now passing.

@liferoad liferoad merged commit fd8d146 into apache:master Aug 30, 2024
105 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug]: Using GcsIO in Python with a DoFn fails
2 participants