[BEAM-5392] group by key non merging window optimalization #49

…ynthetic data to Kafka topic

This prints a warning when checkpointing is disabled and pipelines are run with the FlinkRunner which contain UnboundedSources. Most UnboundedSources rely on checkpoint acknowledgement, e.g. PubSubIO.

…ython connector Write

…ogle-api-core 1.7.0

…of null constructor in ByteArrayShufflePosition

- MetricKey - MetricName - MetricQueryResults - MetricsContainerStepMap

…ommit java

[BEAM-5091] New wordcount integration test on DataflowRunner in Python 3

… to 1.9.12

…s are used without checkpointing

Also resolve two related TODOs.

…e wordcount.py work with attempted metrics

…Python.

- do 10 retries to fix "container not found" error - use later docker image to avoid possible problem

…ython's optimizer on Flink.

`.deflate` files are quite common in Hadoop and also supported by TensorFlow in TFRecord file format. Moreover, `.deflate` is already supported since 0.6.0 by the Java SDK (see BEAM-1518).

…r startup

* [BEAM-6569] Remove a flaky test from a precommit suite. * Remove duplicate mentions of apache_beam.io.gcp.gcsfilesystem_test, apache_beam.io.hdfs_integration_test, apache_beam.io.range_trackers_test, and once again apache_beam.io.gcp.gcsio_test.

Update dependencies in base_image_requirements.txt to sync with setup.py

Update minimal version of futures due to transitive requirement in google-api-core 1.7.0

…irtualenv; use SSH URL

… GroupingTables

…ty check

…orker

…, with Jiras filed

…hen we can

…s is OK

…upStart

… worker

…tory

…bugCapture

…verride

…aflow worker

[BEAM-6574] Inline description of CSV formats

…ed Guava, with exceptions

…prone/checkstyle/findbugs on precommit java

…to Get Metrics API

[BEAM-5315] Python 3 port bigquery and pubsub related io modules

* Adding the gcp_kms_key option to Python SDK. * fix typo

… for Beam SQL, fix errors

Avoid taking abs(MIN_INT) which is undefined.

The `JobInvocation` only reported the top-most message in the stack trace. This changes the reporting to pass on the root cause instead. Unblocks several Python PVR tests which rely on exception reporting.

…SDK dependencies installed.

…n Java SDK User counters

…ure in Flink portable pipelines

… CompressedFile

…the Java SDK

[BEAM-6532] Regenerate BQ client to match apitools==0.5.26 and fix BQ io to work on Python 3

[BEAM-5315] Partially port Avroio module

Work around issue in dill for wordcount example save_main_session

Python 3 fixes for GCP auth and Avro IO

) * Examine func return types when generating shims starcgenx.Extractor should examine both function input and output when extracting types from a package. Otherwise, the generated code may miss certain package imports. * Refactor extractParameters Rename extractParameters to extractFromSignature, which calls extractFromTuple using both params and results.

Use Dataflow dev container images that have current versions of Beam SDK dependencies installed.

…iner Update Dataflow container, select correct Python 3 container

… to 1.9.13

[BEAM-3306] Add first pass coder registry for Go

…r the case of temporary tables

…transform expansion protocol."" This reverts commit a65f1c4.

Fix BigQuery RowAsDictJsonCoder output type in Python 3

Remove EXPERIMENTAL_BEAM_PY3 and add a warning for Python 3 execution

This reverts commit afe1a84.

…o shaded jar

* [BEAM-6291] Generic schema for BQ load tests * [BEAM-6291] Added pipeline option to check if metrics are required * [BEAM-6291] Common environment variable to disable load tests. * [BEAM-6291] Refactored BQ and Metrics load tests utils. Added missing documentation and pipelineoption.

…PreCommit

…metrics start,process,finish in the Java SDK

[BEAM-5091] Run integration test in Python 3 in ValidatesContainer test

…unner. (apache#7722)

[BEAM-6593] Python 3 IT on DirectRunner in PreCommit

Update starcgen for new types

Fix broken link in BigQuery Python connector docs

…iment (apache#7728) * There is no need to predicate WindmillServiceStreamingRpcBatchLimitFactory on the windmill_service_streaming_rpc_batching experiment. That experiment is now fully rolled out. * Fixed argument to streamingEngineEnabled.

* Revert "Revert "Merge pull request apache#7316 [BEAM-6269] Cross-SDK transform expansion protocol."" This reverts commit a65f1c4. * Use ints rather than strings for language boundary crossing. * Revert "Create a standard coder for utf-8 encoded strings." This reverts commit afe1a84. * Make grpc dependency optional.

Setting this flag should prevent the addition of use_multiple_sdk_containers experiment.

…est in Javadoc

Fix a bug in use of no_use_multiple_sdk_containers experiment flag.

…Hadoop InputFormat integration test in Javadoc

When encoding Key Column values, if the column value is unspecified, assume that the value is null.

Before execution, the Flink Runner prepares all necessary files, in order for them to be shipped to the remote cluster. Due to a recent regression via BEAM-2873, the preparation is performed after the execution environment has been initialized. This skips the preparation phase and results in invalid files being shipped.

…mp directory There should be a sensible default, so users don't have to configure it by default.

Use the same noescape that the Go standard library in ioutilx

Remove noisy logging.

* [BEAM-6540] Report Throttling due to Streaming RPC Quota

…s early enough in Flink Runner

Fix the matiching uses of non-vendored guava and adds suppressions for cases when guava is part of the components API.

* BEAM[6571] Adds flags for streaming engine

…lugin

…idempotent

…automation

Add string method to unit status

GCS operations refactored to support objects with KMS keys. Beam will not set KMS keys on GCS objects. Users can utilize bucket default keys. Also adds --dataflowKmsKey, which is passed to DataflowRunner. Details: - GCS copy operation reimplemented as rewrite - allowing copies of objects using KMS keys (source or dest). - Rewrite also supports copying across regions and storage classes. https://cloud.google.com/storage/docs/json_api/v1/objects/rewrite - Introduces --dataflowKmsKey flag, which should apply to Dataflow pipeline state. - If creating a new bucket for gcpTempLocation, sets the bucket default key to --dataflowKmsKey. - New integration tests: - GcsKmsKeyIT - pipeline with GCS sink using --dataflowKmsKey and gcpTempLocation. - GcsUtilIT - tests multi-part rewrite against prod GCS

Adding a gradle target and a test class for running ValidatesRunner tests with the Java Reference Runner. Also had to add a dependency to base_image_requirements.txt to get it to run properly. Note that this was also causing an error when running Flink's VR tests with Docker.

Website: Update built-in IO table

[INFRA-17335] Check more Python versions in inventory job

[BEAM-5959] Adding the dataflow_kms_key option to Python SDK

[BEAM-6625] Reducing test size to simplify running it locally.

… of non-vendored Guava

…jenkins This allows to publish FindBugs reports in XML format on Jenkins and in HTML format on other environments.

Co-Authored-By: TheNeuralBit <hulettbh@gmail.com>

…annotation

…ADME

…r Java Reference Runner.

[BEAM-6603] Fix Python logging in setup.py

State that Datastore IO is not supported in Streaming scenario.

…ails if array/map has null values * Add test for encoding array/map with nulls * Fix bug where null values in Row arrays and maps could not be encoded * Move RowCoderTests to same location as other coder tests * Deprecate array and map schema functions with nullable parameters

[BEAM-5959] KMS support for BigQuery - build.gradle only

…e#7756) * [BEAM-6619] Add PostCommit sutie for Py3 ITs on DataflowRunner

[BEAM-5959] KMS support for BigQuery - build.gradle only - fix

Label Dataflow jobs with SDK name including Python major / minor version

[BEAM-6621] Basic Bigquery IO tests on Python 3 PostCommit

…ndard coder in Python.

…che#6181

Adds a dependency on `zstd-jni` which declares no transitive dependencies, but otherwise just fills out enum values to expose the ZStandard support that already exists in commons-compress.

… support for Java SDK

…e inputs

Generic types are de-facto @nullable so this annotation is redundant and probably only existed to make FindBugs happy.

…zed side inputs between Spark tasks

…n encoding Key columns.

Adds withKmsKey() methods to reads and writes. Affects tables created tables by: streaming writes, batched writes, query temporary tables, table copies (for very large batched writes).

[BEAM-5667] Use BeamDeprecationWarning in BQ client code.

[BEAM-6636] Remove --kmsKey usage

…y3 limitations. (apache#7773) * Restrict pyvcf dependency to Python 2 and link Jira issues tracking Py3 limitations. * Skip all VCF IO tests so that we don't need to install pyvcf on Python 3. * Don't import VCF on Python 2.

…get()" This reverts commit 72627eb.

…unner

…for kryo registration in Spark runner

[BEAM-5959] KMS support for BigQuery

…transforms (apache#7709) Multi-language support in DefaultJobBundleFactory

…f manual classpath

…r interval >= 25 DAY [BEAM-6632] Fixes integer overflow for interval >= 25 DAY

…iption

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-5392] group by key non merging window optimalization #49

[BEAM-5392] group by key non merging window optimalization #49

Commits on Jan 30, 2019

Commits on Jan 31, 2019

Commits on Feb 1, 2019

Commits on Feb 2, 2019

Commits on Feb 4, 2019

Commits on Feb 5, 2019

Commits on Feb 6, 2019

Commits on Feb 7, 2019

Commits on Feb 8, 2019

Commits on Feb 9, 2019

Commits on Feb 10, 2019

Commits on Feb 11, 2019

Commits on Feb 13, 2019

[BEAM-5392] group by key non merging window optimalization #49

Are you sure you want to change the base?

[BEAM-5392] group by key non merging window optimalization #49

Commits on Jan 30, 2019

Commits on Jan 31, 2019

Commits on Feb 1, 2019

Commits on Feb 2, 2019

Commits on Feb 4, 2019

Commits on Feb 5, 2019

Commits on Feb 6, 2019

Commits on Feb 7, 2019

Commits on Feb 8, 2019

Commits on Feb 9, 2019

Commits on Feb 10, 2019

Commits on Feb 11, 2019

Commits on Feb 13, 2019