-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BEAM-143] add test for UnboundedSourceWrapper #69
Conversation
Now, we initialize the UnboundedSourceReader at runtime which requires us to keep a copy of the PipelineOptions. This should be fine here because we are at the lowest point of the execution stack.
The test ensures serialization and execution of the wrapper works as expected.
* | ||
*</p> | ||
* For now we support non-parallel, not checkpointed sources. | ||
* For now we only support non-parallel sources. | ||
* */ | ||
public class UnboundedSourceWrapper<T> extends RichSourceFunction<WindowedValue<T>> implements Triggerable { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like this is a wrapper for Read.Unbounded
not UnboundedSource
-- maybe should rename to UnboundedReadWrapper
? (Or ReadUnboundedWrapper
)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good idea.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it really is a wrapper for UnboundedSource
. In my view Read.Unbounded
is just the API construct that itself contains an UnboundedSource
. Said source is then handled by the UnboundedSourceWrapper
at runtime.
Peppered a few comments, but looks great to me. Glad to have the chance to get my eyes on some Flink code! |
er, +1 |
…riptors apache#61 [euphoria-flink] Do not hold states in memory
Update version to 2.45.1
So far the
UnboundedSourceWrapper
has not been tested systematically which led to a bunch of regression issues. This PR fixes the UnboundedSourceWrapper and adds a test case. The test case ensures that the wrapper works as expected. In particular, a full integration test is run to check whether serialization and instantiation of the wrapper works in a cluster setup.