-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Move stagingLocation and tempLocation to GcsOptions #31
Conversation
Hm... Can you argue more why this is the right change?
|
The reason is BigQueryIO needs a tempLocation option across runners. Another option is to pull both of them to a new PipelineOption. |
Sounds like we might need a job-wide temporary location. I think this needs some discussion. |
This one is tough -- the Would be nice to be able to reuse the one from DataflowPipelineOptions if such exists. |
f95b969
to
a78a806
Compare
Don't close data channels from environment sessions
…tion with Flink executor
apache#31 [euphoria-core] Implementation of accumulator API
…tion with Flink executor
* feat: add new data record read metrics Adds the following metrics: - Initial partition from created to scheduled state time (in ms) - Data record commit timestamp to read time (in ms) - Data record read time to emitted time (in ms) * feat: refine logging Adds the partition token to all possible logging. Add temporary metric of data record count as well. * chore: spotless apply
This will allow BigQueryOptions to extend GcsOptions, and BigQueryIO can access them through BigQueryOptions.