[BEAM-5392] group by key non merging window optimalization #49
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dmvk
reviewed
Oct 26, 2018
| w, | ||
| windowingStrategy.getWindowFn().getOutputTime(windowedValue.getTimestamp(), w)); | ||
| WindowedValue<K> keyWinValue = | ||
| WindowedValue.of(key, timestamp, w, windowedValue.getPane()); //todo predavat pane |
dmvk
reviewed
Oct 26, 2018
|
|
||
| @Override | ||
| public int compare(CompositeKey o1, CompositeKey o2) { | ||
| int keyCompare = SignedBytes.lexicographicalComparator().compare(o1.getKey(), o2.getKey()); |
dmvk
reviewed
Oct 26, 2018
| * @param <K> type of key iterator emits | ||
| * @param <V> type of value iterator emits | ||
| */ | ||
| static class ReduceByKeyIterator<K, V, W extends BoundedWindow> |
dmvk
reviewed
Oct 26, 2018
| Iterators.transform(itr, t2 -> t2._1.withValue(KV.of(t2._1().getValue(), t2._2()))); | ||
| } | ||
|
|
||
| public static class CompositeKeyComparator implements Comparator<CompositeKey>, Serializable { |
mareksimunek
force-pushed
the
simunek/groupByKeyNonMergingWin/BEAM-5392
branch
from
33ef415
to
41d4b8f
Compare
mareksimunek
force-pushed
the
simunek/groupByKeyNonMergingWin/BEAM-5392
branch
from
ab8e5e9
to
997caf4
Compare
mareksimunek
force-pushed
the
simunek/groupByKeyNonMergingWin/BEAM-5392
branch
2 times, most recently
from
ceaee54
to
b95a484
Compare
mareksimunek
force-pushed
the
simunek/groupByKeyNonMergingWin/BEAM-5392
branch
6 times, most recently
from
d66cbeb
to
5189810
Compare
…ynthetic data to Kafka topic
This prints a warning when checkpointing is disabled and pipelines are run with the FlinkRunner which contain UnboundedSources. Most UnboundedSources rely on checkpoint acknowledgement, e.g. PubSubIO.
This prints a warning when checkpointing is disabled and pipelines are run with the FlinkRunner which contain UnboundedSources. Most UnboundedSources rely on checkpoint acknowledgement, e.g. PubSubIO.
…ython connector Write
…ogle-api-core 1.7.0
…of null constructor in ByteArrayShufflePosition
- MetricKey - MetricName - MetricQueryResults - MetricsContainerStepMap
… support for Java SDK
Generic types are de-facto @nullable so this annotation is redundant and probably only existed to make FindBugs happy.
…zed side inputs between Spark tasks
mareksimunek
force-pushed
the
simunek/groupByKeyNonMergingWin/BEAM-5392
branch
from
835132f
to
5a7e8fa
Compare
…n encoding Key columns.
Adds withKmsKey() methods to reads and writes. Affects tables created tables by: streaming writes, batched writes, query temporary tables, table copies (for very large batched writes).
[BEAM-5667] Use BeamDeprecationWarning in BQ client code.
[BEAM-6636] Remove --kmsKey usage
…y3 limitations. (apache#7773) * Restrict pyvcf dependency to Python 2 and link Jira issues tracking Py3 limitations. * Skip all VCF IO tests so that we don't need to install pyvcf on Python 3. * Don't import VCF on Python 2.
…get()" This reverts commit 72627eb.
…for kryo registration in Spark runner
[BEAM-5959] KMS support for BigQuery
…transforms (apache#7709) Multi-language support in DefaultJobBundleFactory
…f manual classpath
…r interval >= 25 DAY [BEAM-6632] Fixes integer overflow for interval >= 25 DAY
mareksimunek
force-pushed
the
simunek/groupByKeyNonMergingWin/BEAM-5392
branch
from
5a7e8fa
to
023f31d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Follow this checklist to help us incorporate your contribution quickly and easily:
[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.It will help us expedite review of your Pull Request if you tag someone (e.g.
@username) to look at it.Post-Commit Tests Status (on master branch)