Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BEAM-4520] Warn when UnboundedSources are used without checkpointing #7673

Merged
merged 2 commits into from
Jan 31, 2019
Merged

[BEAM-4520] Warn when UnboundedSources are used without checkpointing #7673

merged 2 commits into from
Jan 31, 2019

Conversation

mxm
Copy link
Contributor

@mxm mxm commented Jan 30, 2019

This prints a warning when checkpointing is disabled and pipelines are run with
the FlinkRunner which contain UnboundedSources. Most UnboundedSources rely on
checkpoint acknowledgement, e.g. PubSubIO.

Post-Commit Tests Status (on master branch)

Lang SDK Apex Dataflow Flink Gearpump Samza Spark
Go Build Status --- --- --- --- --- ---
Java Build Status Build Status Build Status Build Status
Build Status
Build Status		 Build Status Build Status Build Status
Python Build Status --- Build Status
Build Status		 Build Status --- --- ---

@mxm
[BEAM-4520] Warn when UnboundedSources are used without checkpointing
fe27329 
This prints a warning when checkpointing is disabled and pipelines are run with
the FlinkRunner which contain UnboundedSources. Most UnboundedSources rely on
checkpoint acknowledgement, e.g. PubSubIO.
@mxm
[BEAM-4520] Add note on checkpointing of UnboundedSources
8803821 
This prints a warning when checkpointing is disabled and pipelines are run with
the FlinkRunner which contain UnboundedSources. Most UnboundedSources rely on
checkpoint acknowledgement, e.g. PubSubIO.
@mxm mxm requested a review from chamikaramj January 30, 2019 17:08
@mxm
Copy link
Contributor Author

mxm commented Jan 30, 2019

Run Python PreCommit

1 similar comment
@mxm
Copy link
Contributor Author

mxm commented Jan 30, 2019

Run Python PreCommit

@mxm
Copy link
Contributor Author

mxm commented Jan 30, 2019

Python PreCommit is flaky, unrelated to this PR:

18:51:43 * What went wrong:
18:51:43 Execution failed for task ':beam-sdks-python:testPy3Gcp'.
18:51:43 > Process 'command 'sh'' finished with non-zero exit value 1

@chamikaramj
Copy link
Contributor

LGTM. Thanks.

@mxm mxm merged commit 42d0918 into apache:master Jan 31, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chamikaramj chamikaramj Awaiting requested review from chamikaramj

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mxm @chamikaramj