[BEAM-1] Removal of a few windmill-related protos that aren't Beam-specific #2

davorbonaci · 2016-02-29T19:02:20Z

This pull request removes windmill.proto and windmill_service.proto.

kennknowles · 2016-03-01T16:19:52Z

LGTM

Fix more failing tests on Flink

Fixes examples

DoFnInvokers work for Splittable DoFn.

Syncing from original

…t-check [BEAM-1726] Obey watermark hold in DoFnOperato.processWatermark2()

Write minimal interface for Flink HarnessManager and related classes. This defines a minimal interface for a Flink HarnessManager, as well as the related interfaces of EnvironmentSession and ArtifactSource. Add beam-runners-java-fn-execution to the flink-runner's maven and gradle dependencies for planned work on integrating with the beam portability framework. This will be necessary to interact with protobufs related to the artifact API while implementing ArtifactService and related classes.

Add pipeline options to the Python ULR prepare job request.

Fix the code that uses Long key whereas there is no key in the PCollection

#! [euphoria-kafka] Properly quit read loop when interrupted

Add project configuration for lint warnings and disable dep-ann in SQL.

[BEAM-4176] Add the ability to allow for runners to register native transforms.

fixup! Simplify website publishing logic

…te_Publish

…_Publish (#6531)

…eams data from twitter * # This is a combination of 2 commits. # This is the 1st commit message: Java PreCommit failure fix spotless failure fix Java PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit refix Java_Examples_Dataflow PreCommit fix build failure corrected Spotless check Spotless check reorganizing pipeline delete the unused folder Revert "Delete build.gradle" This reverts commit c39a4e4 Delete build.gradle don't need this file adding comments and java docs, and removing unneeded dependencies. Linting the project and making some stuff private Reorganized and redefined to logic as per standard beam IO structure. Lint the files. Added changes for making the implementation more streamlined and understandable Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: # This is a combination of 15 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: Added changes for making the implementation more streamlined and understandable # This is the commit message #3: Lint the files. # This is the commit message #4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message #5: Linting the project and making some stuff private # This is the commit message #6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message #7: delete the unused folder # This is the commit message #8: reorganizing pipeline # This is the commit message #9: Spotless check # This is the commit message #10: Spotless check # This is the commit message #11: build failure corrected # This is the commit message #12: Java_Examples_Dataflow PreCommit fix # This is the commit message #13: Java_Examples_Dataflow PreCommit refix # This is the commit message #14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #15: Java_Examples_Dataflow PreCommit assign nullable correctly * # This is a combination of 2 commits. # This is the 1st commit message: # This is a combination of 2 commits. # This is the 1st commit message: Java PreCommit failure fix spotless failure fix Java PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit refix Java_Examples_Dataflow PreCommit fix build failure corrected Spotless check Spotless check reorganizing pipeline delete the unused folder Revert "Delete build.gradle" This reverts commit c39a4e4 Delete build.gradle don't need this file adding comments and java docs, and removing unneeded dependencies. Linting the project and making some stuff private Reorganized and redefined to logic as per standard beam IO structure. Lint the files. Added changes for making the implementation more streamlined and understandable Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: # This is a combination of 15 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: Added changes for making the implementation more streamlined and understandable # This is the commit message #3: Lint the files. # This is the commit message #4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message #5: Linting the project and making some stuff private # This is the commit message #6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message #7: delete the unused folder # This is the commit message #8: reorganizing pipeline # This is the commit message #9: Spotless check # This is the commit message #10: Spotless check # This is the commit message #11: build failure corrected # This is the commit message #12: Java_Examples_Dataflow PreCommit fix # This is the commit message #13: Java_Examples_Dataflow PreCommit refix # This is the commit message #14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #15: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #2: # This is a combination of 3 commits. # This is the 1st commit message: Java PreCommit failure fix spotless failure fix Java PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit refix Java_Examples_Dataflow PreCommit fix build failure corrected Spotless check Spotless check reorganizing pipeline delete the unused folder Revert "Delete build.gradle" This reverts commit c39a4e4 Delete build.gradle don't need this file adding comments and java docs, and removing unneeded dependencies. Linting the project and making some stuff private Reorganized and redefined to logic as per standard beam IO structure. Lint the files. Added changes for making the implementation more streamlined and understandable Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: # This is a combination of 15 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: Added changes for making the implementation more streamlined and understandable # This is the commit message #3: Lint the files. # This is the commit message #4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message #5: Linting the project and making some stuff private # This is the commit message #6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message #7: delete the unused folder # This is the commit message #8: reorganizing pipeline # This is the commit message #9: Spotless check # This is the commit message #10: Spotless check # This is the commit message #11: build failure corrected # This is the commit message #12: Java_Examples_Dataflow PreCommit fix # This is the commit message #13: Java_Examples_Dataflow PreCommit refix # This is the commit message #14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #15: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #3: # This is a combination of 16 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: Added changes for making the implementation more streamlined and understandable # This is the commit message #3: Lint the files. # This is the commit message #4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message #5: Linting the project and making some stuff private # This is the commit message #6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message #7: delete the unused folder # This is the commit message #8: reorganizing pipeline # This is the commit message #9: Spotless check # This is the commit message #10: Spotless check # This is the commit message #11: build failure corrected # This is the commit message #12: Java_Examples_Dataflow PreCommit fix # This is the commit message #13: Java_Examples_Dataflow PreCommit refix # This is the commit message #14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #15: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #16: Java PreCommit assign nullable correctly Java PreCommit assign nullable correctly spotless failure fix Java PreCommit failure fix correcting the if checks cleaning up and adding readme spotless fixed readme fixed and compileJava fix compileJava fix compileJava fix now spotless fix now Java PreCommi fix Java PreCommit fix # This is a combination of 16 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: Added changes for making the implementation more streamlined and understandable # This is the commit message #3: Lint the files. # This is the commit message #4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message #5: Linting the project and making some stuff private # This is the commit message #6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message #7: delete the unused folder # This is the commit message #8: reorganizing pipeline # This is the commit message #9: Spotless check # This is the commit message #10: Spotless check # This is the commit message #11: build failure corrected # This is the commit message #12: Java_Examples_Dataflow PreCommit fix # This is the commit message #13: Java_Examples_Dataflow PreCommit refix # This is the commit message #14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #15: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #16: Java PreCommit assign nullable correctly Java PreCommit assign nullable correctly spotless failure fix Java PreCommit failure fix correcting the if checks cleaning up and adding readme spotless fixed readme fixed and compileJava fix compileJava fix compileJava fix now spotless fix now Java PreCommi fix Java PreCommit fix # This is a combination of 3 commits. # This is the 1st commit message: Java PreCommit failure fix spotless failure fix Java PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit assign nullable correctly Java_Examples_Dataflow PreCommit refix Java_Examples_Dataflow PreCommit fix build failure corrected Spotless check Spotless check reorganizing pipeline delete the unused folder Revert "Delete build.gradle" This reverts commit c39a4e4 Delete build.gradle don't need this file adding comments and java docs, and removing unneeded dependencies. Linting the project and making some stuff private Reorganized and redefined to logic as per standard beam IO structure. Lint the files. Added changes for making the implementation more streamlined and understandable Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: # This is a combination of 15 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: Added changes for making the implementation more streamlined and understandable # This is the commit message #3: Lint the files. # This is the commit message #4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message #5: Linting the project and making some stuff private # This is the commit message #6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message #7: delete the unused folder # This is the commit message #8: reorganizing pipeline # This is the commit message #9: Spotless check # This is the commit message #10: Spotless check # This is the commit message #11: build failure corrected # This is the commit message #12: Java_Examples_Dataflow PreCommit fix # This is the commit message #13: Java_Examples_Dataflow PreCommit refix # This is the commit message #14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #15: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #3: # This is a combination of 16 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: Added changes for making the implementation more streamlined and understandable # This is the commit message #3: Lint the files. # This is the commit message #4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message #5: Linting the project and making some stuff private # This is the commit message #6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message #7: delete the unused folder # This is the commit message #8: reorganizing pipeline # This is the commit message #9: Spotless check # This is the commit message #10: Spotless check # This is the commit message #11: build failure corrected # This is the commit message #12: Java_Examples_Dataflow PreCommit fix # This is the commit message #13: Java_Examples_Dataflow PreCommit refix # This is the commit message #14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #15: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #16: Java PreCommit assign nullable correctly Java PreCommit assign nullable correctly spotless failure fix Java PreCommit failure fix correcting the if checks cleaning up and adding readme spotless fixed readme fixed and compileJava fix compileJava fix compileJava fix now spotless fix now Java PreCommi fix Java PreCommit fix # This is a combination of 16 commits. # This is the 1st commit message: Added a connector that streams data from twitter using a Standard Twitter app. # This is the commit message #2: Added changes for making the implementation more streamlined and understandable # This is the commit message #3: Lint the files. # This is the commit message #4: Reorganized and redefined to logic as per standard beam IO structure. # This is the commit message #5: Linting the project and making some stuff private # This is the commit message #6: adding comments and java docs, and removing unneeded dependencies. # This is the commit message #7: delete the unused folder # This is the commit message #8: reorganizing pipeline # This is the commit message #9: Spotless check # This is the commit message #10: Spotless check # This is the commit message #11: build failure corrected # This is the commit message #12: Java_Examples_Dataflow PreCommit fix # This is the commit message #13: Java_Examples_Dataflow PreCommit refix # This is the commit message #14: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #15: Java_Examples_Dataflow PreCommit assign nullable correctly # This is the commit message #16: Java PreCommit assign nullable correctly Java PreCommit assign nullable correctly spotless failure fix Java PreCommit failure fix correcting the if checks cleaning up and adding readme spotless fixed readme fixed and compileJava fix compileJava fix compileJava fix now spotless fix now Java PreCommi fix Java PreCommit fix Final Commit with all changes Added unit test adding examples for usage usage for TwitterIO added and Java PreCommit failure fix Spotless PreCommit failure fix * Unit test for multiple config added, and beautification * Spotless apply fixed * Removing redundant comments * Removing newly added test * adding newly added test back

# This is the 1st commit message: restructure javadocs a big to keep context close to code samples # The commit message apache#2 will be skipped: # fix typo in javadocs

Update li_trunk branch to no longer use internal samza-versions

…he Beam Playground * Implement Beam Playground gradle tasks * Fix trailing whitespace * Remove format from precommit * Fix order of includes and add descriptions * [BEAM-13008] add generated files and fix analysis issues (#2) * Reconcile errors with playgroundPrecommit output Co-authored-by: Aydar Farrakhov <stranniknm@gmail.com>

… for the Beam Playground * Implement Beam Playground gradle tasks * Fix trailing whitespace * Remove format from precommit * Fix order of includes and add descriptions * [BEAM-13008] add generated files and fix analysis issues (apache#2) * Reconcile errors with playgroundPrecommit output Co-authored-by: Aydar Farrakhov <stranniknm@gmail.com>

commit 6c954ca7106e67c0834b0e74926c376f02d5f811 Merge: e237c2fafb a1bfd088cc Author: Pablo <pabloem@users.noreply.github.com> Date: Tue Nov 9 10:45:26 2021 -0800 Merge pull request #15868 from msbukal/PIPEWHEEL Minor HL7v2IO improvements commit e237c2fafb28223e607ecd41a6d59b23dab43ad7 Author: Milena Bukal <msbukal@google.com> Date: Tue Nov 9 13:00:45 2021 -0500 Improve FhirIO LRO Counters + minor fixes (#15921) * Improve LRO reporting Log the operation status, not just # of success/failures. Additionally added counters for this. * Increased polling time for operations Polling every 500ms is excessive, 15 seconds should be enough for expected time for an operation to complete. * Remove unsupported content structure for FHIR imports. FhirIO.Import specifically formats input JSON into NDJSON (even for pretty input) and then writes multiple resources to a single file. Therefore, only RESOURCE or BUNDLE content type can be specified. There is no functionality lost here, as pretty JSON can still be passed as input and will be transformed before the import operation begins. * Nit cleanups + missed increment. Increment counter for failure to start operations too (easy to miss this log and otherwise the job says it completed successfully). commit 57e331bbb08d16b280cbf679be458e98a9b781a4 Author: Aydar Zainutdinov <aydar.zaynutdinov@akvelon.com> Date: Tue Nov 9 20:48:02 2021 +0300 Merge pull request #15804 from [BEAM-13109][Playground] Add processing of timeout for RunCode API method * [BEAM-13109][Playground] add timeout processing for processCode() method; add remote cache option to setupCache() method; * [BEAM-13109][Playground] fix tests after merge with master * [BEAM-13109][Playground] added new status: Status_STATUS_PREPARING added new status: Status_STATUS_PREPARATION_ERROR added preparation logic to processCode() method * [BEAM-13109][Playground] added TODO; renaming; * [BEAM-13109][Playground] regenerate pb files commit 39bd3a4074316f31114e93bb489c85ffebb9a0dc Merge: 7ebccbcea5 591f00ee8f Author: Andrew Pilloud <apilloud@users.noreply.github.com> Date: Tue Nov 9 08:56:32 2021 -0800 Merge pull request #15886: [BEAM-13056] Expose FieldAccess in DoFnSchemaInformation commit 7ebccbcea53d0b99edc043b72a557aa155b4742e Author: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Date: Tue Nov 9 11:12:00 2021 -0500 [BEAM-11097] Create hook to enable cross-bundle side input caching (#15896) commit 591f00ee8fe60a2937456cd3ed0bf40702fdf402 Author: Andrew Pilloud <apilloud@google.com> Date: Wed Nov 3 14:33:18 2021 -0700 [BEAM-13056] Expose FieldAccess in DoFnSchemaInformation commit 2f2e4fac9c9814ecba2e34d3700414fc4bc9c85d Merge: cb1d5dec39 ea1bf40ef6 Author: Kyle Weaver <kcweaver@google.com> Date: Mon Nov 8 14:28:15 2021 -0800 Merge pull request #15922 from ibzib/es-flake [BEAM-5172] Temporary ignore testSplit and testSizes tests waiting fo… commit ea1bf40ef61b77bfd6a4d0e73fab9ae147a70f69 Author: Etienne Chauchot <echauchot@apache.org> Date: Fri Sep 3 09:49:04 2021 +0200 [BEAM-5172] Temporary ignore testSplit and testSizes tests waiting for a fix because they are flaky. commit cb1d5dec39306c1017832a1fca4ea0e288d841ae Author: Aydar Farrakhov <stranniknm@gmail.com> Date: Mon Nov 8 23:46:00 2021 +0300 Merge pull request #15813 from [BEAM-13071] [Playground] Update run code statuses and add notification system * [BEAM-13071]: add notification system * [BEAM-13071]: update run code statuses with new protobuf * [BEAM-13071] fix generated files * [BEAM-13071] fix generated files * [BEAM-13071] fix dart analysis * [BEAM-13071] remove duplicated run status cases * [BEAM-13071] fix licencess commit f068a62e5498abf5e53945c992fd2a084ab14432 Merge: 5e44a133ce 24205850ef Author: Pablo <pabloem@users.noreply.github.com> Date: Mon Nov 8 12:44:18 2021 -0800 Merge pull request #15879 from [BEAM-13155][Playground] Update the processing of error during run code [BEAM-13155][Playground] Update the processing of error during run code commit 5e44a133ce69c95aa434536b73ae6f757c243cce Author: Milena Bukal <msbukal@google.com> Date: Mon Nov 8 15:19:09 2021 -0500 Merge pull request #15873 from [BEAM-13181] Remove Sharding from FhirIO.Import * Test FhirIO improvements * Remove batching from FhirIO.Import * Minor cleanup * Fix tmpGcsPath input commit 84d57ca36294f1bc29d22a4ac4282a458999946c Merge: 79496b2ff9 2a94534a3d Author: Alexey Romanenko <33895511+aromanenko-dev@users.noreply.github.com> Date: Mon Nov 8 15:44:49 2021 +0100 Merge pull request #15914: [BEAM-13157] add regression test for hadoop configuration on ParquetIO.Parse commit 79496b2ff9f65c4584fa1a7a8e9630ffd6d16222 Merge: c04a2a8fef a9d9871512 Author: reuvenlax <relax@google.com> Date: Sat Nov 6 11:31:53 2021 -0700 Merge pull request #15913: [BEAM-2791] remove spammy log statement commit 2a94534a3d634c9c14f210bf24f02418157dad3f Author: Michael Luckey <25622840+adude3141@users.noreply.github.com> Date: Sat Nov 6 17:41:24 2021 +0100 [BEAM-13157] add regression test for hadoop configuration on ParquetIO.Parse commit c04a2a8fef45a7852491988567fee62edfd25d0f Merge: a3bb58dbd4 ba6aa4eb92 Author: Andrew Pilloud <apilloud@users.noreply.github.com> Date: Fri Nov 5 18:23:31 2021 -0700 Merge pull request #15912: [BEAM-13080] Fix number of default keys in Reshuffle commit ba6aa4eb92c6f8c7367b32ac47b920a68ce58aab Author: InigoSJ <joseinigo@google.com> Date: Fri Nov 5 23:44:22 2021 +0000 [BEAM-13080] Fix number of default keys commit a9d9871512e3ea42c46bb5dada247403ef762db3 Author: Reuven Lax <relax@google.com> Date: Fri Nov 5 16:33:23 2021 -0700 remove log statement commit f2e660bd5095f93f5867f3781e8a16798677e378 Author: InigoSJ <joseinigo@google.com> Date: Fri Nov 5 22:13:33 2021 +0000 [BEAM-13080] Fix number of default keys commit a3bb58dbd4fc6a59f93f69f9ab1980f8232b6e82 Author: Robert Burke <lostluck@users.noreply.github.com> Date: Fri Nov 5 13:17:18 2021 -0700 [Go SDK] Go SDK Exits Experimental (#15894) commit e668460f61540638fb29e05997087b56ebcee4f3 Author: Ritesh Ghorse <riteshghorse@gmail.com> Date: Fri Nov 5 16:15:24 2021 -0400 [BEAM-11217] Metrics Query filtering for DoFn metrics. (#15887) commit f37b6c72c14bc19574d5cf403f11bf49648e4a54 Author: Moritz Mack <mmack@talend.com> Date: Fri Nov 5 18:02:05 2021 +0100 [BEAM-13192] Fix buggy retry tests for AWS SnsIO (#15910) * [BEAM] Fix buggy retry tests for AWS SnsIO * Fix interface commit 56a4e5858b30027a3f9b477bb5889cd1933e72d9 Author: Benjamin Gonzalez <74670721+benWize@users.noreply.github.com> Date: Fri Nov 5 11:00:45 2021 -0600 [BEAM-12566] Implement set_axis for DataFrame and Series (#15773) * [BEAM-12566] Initial implementation set_axis * [BEAM-12566] Implement set_axis for DataFrame and Series * [BEAM-12566] Reject set_axis(axis=index) * [BEAM-12566] Raise NotImplemented on Series.set_axis commit 90a48ee4744c8766f812f4b673a4f1b295c4d8b7 Author: Moritz Mack <mmack@talend.com> Date: Fri Nov 5 15:15:43 2021 +0100 [adhoc] Speedup slow tests for AWS IO modules (#15899) * [adhoc] Speedup slow tests for AWS IO modules * [adhoc] Speedup slow tests for AWS IO modules * Change EmbeddedSqsServer to use dynamic port to avoid competing for resources in tests commit cb5913aa51243ed3fedc6030ad33bf2d52f52bc8 Author: Jiayang Wu <530081999@qq.com> Date: Thu Nov 4 13:51:43 2021 -0700 [BEAM-13081] Fixes a compatible issue of decoding null-value bitmap between JVM coder and Python Coder (#15829) * tests passed * make one line shorter * reformat * reformat again * reformat for the last time Co-authored-by: Jiayang Wu <jiayangw@twitter.com> commit 5e0030007fdcc1aa521c820c9b1f65384dacf454 Merge: 7497a2c466 e4f8f4b58b Author: Lukasz Cwik <lukecwik@gmail.com> Date: Thu Nov 4 11:51:47 2021 -0700 [BEAM-3811] Code clean-up of the CancelleableQueue to not throw InterruptedException on cancel/reset since we are never stuck in a loop here or waiting on a signal. commit 7497a2c4662a826b1e6a809bba3fbaf52f02fac7 Merge: fafe9d1db1 605485686b Author: Chamikara Jayalath <chamikara@apache.org> Date: Thu Nov 4 10:22:18 2021 -0700 Merge pull request #15891: [BEAM-13187] Set filesToStage after full jar resolution. commit fafe9d1db1bc6d9e29bf33bd29dff5509dadc600 Author: reuvenlax <relax@google.com> Date: Thu Nov 4 09:58:51 2021 -0700 Merge pull request #15810: [BEAM-2791] Support low-latency StorageApi sink with no exactly-once guarantees commit 0b47f1b38dd612c1d0730a6dc6391d0438e6a84a Author: Tomo Suzuki <suztomo@google.com> Date: Thu Nov 4 12:06:27 2021 -0400 Merge pull request #15835: [BEAM-11205] Google Libraries BOM 24.0.0 and associated deps * Google Libraries BOM 24.0.0 and associated dependencies * The example Java test to declare bigquery dependency * ApiSurface test to skip non-public annotations * Added Libraries BOM upgrade into the 2.35.0 changelog commit f4888029d13f8a0691b4a3b118045e6f3f36d0dd Merge: 164a4587d1 963cf92c9f Author: Heejong Lee <heejong@gmail.com> Date: Thu Nov 4 08:59:19 2021 -0700 Merge pull request #15692 from ihji/BEAM-13021 [BEAM-13021] Deduplicate Python artifact not only by hash but also by source path. commit 164a4587d120902e9b330035feaf1de61c31861f Merge: 0262dd6a54 1e388f9733 Author: Lukasz Cwik <lukecwik@gmail.com> Date: Thu Nov 4 07:38:56 2021 -0700 [BEAM-13164] Address most of a race condition between instantiation and usage of member variables. commit 0262dd6a5434ffc60c9ea73e700aeb4dad62ff3e Merge: 0faa3b9403 f829123af1 Author: Alexey Romanenko <33895511+aromanenko-dev@users.noreply.github.com> Date: Thu Nov 4 13:20:01 2021 +0100 Merge pull request #15841: [BEAM-11440] Add integration test for KinesisIO using Localstack (aws2) commit 963cf92c9f4572b8ac714ab07fd539a324ee3c57 Author: Heejong Lee <heejong@gmail.com> Date: Wed Nov 3 18:34:40 2021 -0700 s/pathes/paths/ commit 605485686b67e1ddfd3a471e091a55ae33fa03c9 Author: Robert Bradshaw <robertwb@gmail.com> Date: Wed Nov 3 17:22:54 2021 -0700 Defer filesToStage construction until after full jar resolution. commit 0faa3b9403bd9c852075f2baf9b2e1f9d588677f Merge: c79eaf849e 013da5568a Author: Kyle Weaver <kcweaver@google.com> Date: Wed Nov 3 16:32:21 2021 -0700 Merge pull request #15885 from ibzib/wc-twister2 Fix typo: s/spark/twister2 commit c79eaf849ec7764f48c9d4c9012cb1824ab8ec56 Merge: bee60cdc29 3c3bfa36eb Author: Robert Bradshaw <robertwb@google.com> Date: Wed Nov 3 15:48:05 2021 -0700 Merge pull request #15630 Fix Python test example in documentation commit 1e388f9733237746c3cc715495075460aa240e96 Author: Luke Cwik <lcwik@google.com> Date: Wed Nov 3 15:32:30 2021 -0700 [BEAM-13164] Address most of a race condition between instantiation and usage of member variables. There is a race condition being observed where we are passing the `InboundObserver` to the `outboundObserverFactory` here (https://github.com/apache/beam/blob/bee60cdc29d995170d4461692e391683ca7dcafd/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexer2.java#L68). This allows for incoming calls to start before the BeamFnDataGrpcMultiplexer2 is fully initialized meaning that the erroredInstructionIds is accessed before it is initialized. This seems like a general problem with how we connect the inbound and the outbound observer (because of the ClientCallStreamObserver callack that is necessary) that only happens on the SDK harness since there is no need to connect the inbound and outbound observers this way on the ServerCallStreamObserver. commit 013da5568a64e46204a33e58cc2233eae7e0c577 Author: Kyle Weaver <kcweaver@google.com> Date: Wed Nov 3 15:21:13 2021 -0700 Fix typo: s/spark/twister2 commit bee60cdc29d995170d4461692e391683ca7dcafd Merge: 156bfc2dd7 2cfb6a71b2 Author: Pablo <pabloem@users.noreply.github.com> Date: Wed Nov 3 14:27:20 2021 -0700 Merge pull request #15862 from [BEAM-13165] JdbcIO has a single WriteFn underlying all implementations [BEAM-13165] JdbcIO has a single WriteFn underlying all implementations commit 156bfc2dd7df9541205475d1e3b483ca012db2ac Author: Aydar Farrakhov <stranniknm@gmail.com> Date: Wed Nov 3 23:55:54 2021 +0300 Merge pull request #15852 from [BEAM-13102] [Playground] update playground according with the latest design changes * [BEAM-13102] update playground according to the last design changes * [BEAM-13102] fix license * [BEAM-13102] fix licencse * [BEAM-13102] add licences to svg files * [BEAM-13102] add licences to svg files commit 219bca3ca05c8c18cd345d31f713b7d93de55254 Author: Heejong Lee <heejong@gmail.com> Date: Wed Nov 3 12:51:32 2021 -0700 rearrange comparison order commit 82d94b3a98543212922c8dae6eb9ba9072e34976 Author: Robert Burke <lostluck@users.noreply.github.com> Date: Wed Nov 3 12:47:36 2021 -0700 [BEAM-13119] Subdirectory prefix tag for Go SDK (#15881) commit 0d2b05b646025b1b6f30f205ec49f209e79c9024 Merge: 95e9de7891 d5cd15ab4d Author: Michael Luckey <25622840+adude3141@users.noreply.github.com> Date: Wed Nov 3 20:39:43 2021 +0100 Merge pull request #15856: [BEAM-13157] support hadoop configuration on ParquetIO.Parse commit 95e9de7891593fa73d582cb6ba5ac0333b2675ff Author: Fernando Morales <80284146+fernando-wizeline@users.noreply.github.com> Date: Wed Nov 3 12:31:23 2021 -0700 Merge pull request #15784 from [BEAM-8135] - Removing BigQueryResultsPublisher and its references * Removing BigQueryResultsPublisher and its references * Removing bigquerypublisher code from Groovy tests * Removing remaining bigquery references * Applying spotless commit 4f3b5c0f0625b02ef25311afc11df2babd1a155d Author: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Date: Wed Nov 3 15:18:12 2021 -0400 Add window mapping to CHANGES.md (#15871) commit 0f94f961709f6d728bcc950d5966601a4c5bb6b9 Author: Ritesh Ghorse <riteshghorse@gmail.com> Date: Wed Nov 3 15:16:51 2021 -0400 [BEAM-13001] updated CHANGES.md to include msec counter for Go (#15872) commit 069095300fbbb0f98e9e9b732e84d7d572101a17 Author: Aydar Zainutdinov <aydar.zaynutdinov@akvelon.com> Date: Wed Nov 3 22:11:14 2021 +0300 Merge pull request #15839 from [BEAM-13041][Playground] Prepare files with java code * [BEAM-13041][Playground] Add Prepare step as one of the executor's steps; Add Java preparation methods to remove "package ..." string from file; Add Java preparation methods to change "public class" strings to "class" string into file; * [BEAM-13041][Playground] minor fix * [BEAM-13041][Playground] add using of logger; change the package pattern; commit 4af1e52809be122d40e03eb0fb7925db6feec145 Merge: beac28dbdd f41cdf40fa Author: Lukasz Cwik <lukecwik@gmail.com> Date: Wed Nov 3 12:03:14 2021 -0700 [BEAM-13015] Use a network based channel instead of an inmemory one which allows us to more faithfully mirror performance/overhead of the SDK harness. commit beac28dbdd602e11b1a2ab2f1474abb9655c3f4c Merge: 735db247f3 0d172c3e54 Author: Udi Meiri <udim@users.noreply.github.com> Date: Wed Nov 3 11:56:34 2021 -0700 Merge pull request #15880: Performance improvement to PubSubLiteIO to not use a streaming committer to send one commit message commit 2cfb6a71b234c9c56c0aa4b8dedf066d23c9a566 Author: Pablo Estrada <pabloem@apache.org> Date: Wed Nov 3 11:34:45 2021 -0700 Addressing comments commit 735db247f3e03d9fddb9f6d7281c986b60ac683d Merge: 07aaec344a 6a63eb9932 Author: Ning Kang <kawaigin@gmail.com> Date: Wed Nov 3 11:05:23 2021 -0700 [BEAM-13140] Fixes bug in WriteRename class commit f41cdf40fa44aec6d460bddff5afd168ada1fa40 Author: Luke Cwik <lcwik@google.com> Date: Wed Nov 3 10:34:29 2021 -0700 [BEAM-13015] Use a network based channel instead of an inmemory one which allows us to more faithfully mirror performance/overhead of the SDK harness. commit 07aaec344aef38ac7034e65c909ed75893a9d48d Author: Brian Hulette <bhulette@google.com> Date: Wed Nov 3 10:17:39 2021 -0700 [BEAM-13099] Use vendored Calcite 1.28.0 in SQL extensions (#15836) * [BEAM-13099] Drop vendor:calcite-1_26_0 * [BEAM-13099] update library to vendored-calcite-1_28_0 * [BEAM-13099] s/org.apache.beam.vendor.calcite.v1_26_0/org.apache.beam.vendor.calcite.v1_28_0/ * [BEAM-13099] org.apache.beam.vendor.calcite.{v1_26_0->v1_28_0}.org.codehaus.commons.compiler.properties * [BEAM-13099] Adjust test cases for 1.28.0 changes * [BEAM-13099] Fix AggregateCall.create call for https://github.com/apache/calcite/commit/f1da65504e598928cf77aa6a7244552692ae2529 * [BEAM-13099] Add 'provided' dependency for calcite's immutable annotations * [BEAM-13099] Fix checkerframework errors due to new Nullable annotations in Calcite commit 6a63eb9932824bc723a59804343924f8e2ff3ce1 Author: Hector Lagos <hlagosperez@twitter.com> Date: Thu Oct 28 12:33:52 2021 -0400 Use BigQuieryIO.loadProjectId in WriteRename class to create JobRefernce config and Fix side input for WriteRename commit 24205850efc35caff49f8adbbf04f79e625b575c Author: AydarZaynutdinov <aydar.zaynutdinov@akvelon.com> Date: Wed Nov 3 18:56:24 2021 +0300 [BEAM-13155][Playground] add a new status: STATUS_RUN_ERROR; add a new API method GetRunError; add a new sunKey RunError; update of validation processing; commit 48318dd3aedeb208648149eca36165bbfff09e1a Author: Aydar Farrakhov <stranniknm@gmail.com> Date: Wed Nov 3 18:48:51 2021 +0300 Merge pull request #15782 from [BEAM-13034] [Playground] add semantics to playground app * [BEAM-13034]: add semantics to playground app * [BEAM-13034]: add semantics to playground app * [BEAM-13034]: add lang attribute to improve acceesibility * [BEAM-13034] fix dart analysis commit 9bca51f8e808b660b77ee0e96b9ae0eaa6a20967 Author: Svetak Sundhar <svetaksundhar@google.com> Date: Wed Nov 3 11:45:33 2021 -0400 [BEAM-12550] Parallelizable skew Implementation (#15809) * [BEAM-12550] Implement Parallelizable Skew and Kurtosis (Skew implementation) R: @TheNeuralBit * [BEAM-12550] Implement Parallelizable Skew and Kurtosis (Skew implementation) R: @TheNeuralBit * [BEAM-12550] Implement Parallelizable Skew and Kurtosis (Skew implementation) R: @TheNeuralBit * [BEAM-12550] Implement Parallelizable Skew and Kurtosis (Kurtosis implementation) * [BEAM-12550] Implement Parallelizable Skew and Kurtosis (Kurtosis implementation) * [BEAM-12550] Implement Parallelizable Skew and Kurtosis (Kurtosis implementation) * [BEAM-12550] Implement Parallelizable Skew and Kurtosis (Skew implementation) Co-authored-by: svetakvsundhar <svetak.sundhar@gmail.com> commit 9109354e3d6aefecc8b1a41004347ae9962329ab Author: daria.malkova <daria.malkova@akvelon.com> Date: Wed Nov 3 18:36:48 2021 +0300 Merge pull request #15854 from [BEAM-13046][Playground] protobuf supporting examples API * Change protobuf * Regenerate protobuf files with "buf" gradle task * Add todo's to controller * Proto files * Add License to generated proto files * Change the structure of the list of example * Merge, fix precommit task, add licenses commit 7db9d2c8b54df0987e233e1a777b285e64e20b93 Merge: 92ba229336 b97a1c3698 Author: Pablo <pabloem@users.noreply.github.com> Date: Wed Nov 3 08:34:47 2021 -0700 Merge pull request #15814 from [BEAM-13105] [Playground] - add shortcut hint tooltip [BEAM-13105] [Playground] - add shortcut hint tooltip commit 0d172c3e54d7c773671a6484965d6daa442fec15 Author: Daniel Collins <dpcollins@google.com> Date: Wed Nov 3 11:21:06 2021 -0400 Performance improvement to PubSubLiteIO to not use a streaming committer to send one commit message. commit 92ba229336aa9cc4132f498d7049ecf67bc1c3b3 Merge: 2d80f93b7b 8830745867 Author: tvalentyn <tvalentyn@users.noreply.github.com> Date: Tue Nov 2 15:33:32 2021 -0700 [BEAM-12730] Handle corner case for TextIO with custom delimiter #15864 commit a1bfd088cc26c0c9576072c6698447c12b397248 Merge: c4feaff74b 2d80f93b7b Author: Milena Bukal <msbukal@google.com> Date: Tue Nov 2 16:58:51 2021 -0400 Merge branch 'apache:master' into PIPEWHEEL commit 2d80f93b7bce4bc42498dc845e9e411641b9cd90 Merge: a2885d57dc 8cdc527407 Author: Andrew Pilloud <apilloud@users.noreply.github.com> Date: Tue Nov 2 12:17:29 2021 -0700 Merge pull request #15846: [BEAM-13118] Use branch for nightly "tag", tags are immutable commit a2885d57dc30788b272759d39e738c0870ea7d17 Author: Brian Hulette <bhulette@google.com> Date: Tue Nov 2 11:52:44 2021 -0700 Minor: add resource key to presentation materials link (#15867) commit 5ca2647670bd99cfe95537ada8a82b4bf5f6871c Author: Pablo <pabloem@users.noreply.github.com> Date: Tue Nov 2 11:13:09 2021 -0700 Merge pull request #15832 from [BEAM-13149] Changing readWithPartitions to work with longs. Removing untested (non-working) code path * Changing readWithPartitions to work with longs. Removing untested (non-working) code path * Address comments commit 526359b27e61463c5b9b2fa86d9ba23ed5890f3e Author: Pavel Avilov <avilovpavel6@gmail.com> Date: Tue Nov 2 20:42:24 2021 +0300 Merge pull request #15838 from [BEAM-13127] [Playground] Implement TCP listener * Implement TCP listener * Using our own logger Co-authored-by: Pavel Avilov <pavel.avilov> commit 8162fb3c8e7162fb413a3150aef4d83433dbabb6 Author: Aydar Zainutdinov <aydar.zaynutdinov@akvelon.com> Date: Tue Nov 2 20:36:23 2021 +0300 Merge pull request #15842 from [BEAM-13125][Playground] Update documentation for environments * [BEAM-13125][Playground] add comments for all environments files * [BEAM-13125][Playground] update comments * [BEAM-13125][Playground] update comments commit c4feaff74b093152cea2dc2523f85a2dd79d5dd7 Author: Milena Bukal <msbukal@google.com> Date: Tue Nov 2 13:24:31 2021 -0400 Minor HL7v2IO Fixes - Fix comments, HL7v2 import is supported & in GA, but not supported by the HL7v2IO. - Fix unused counter. commit 49d828a9915a858ddc9dd87049ec477601db0ed5 Author: Ritesh Ghorse <riteshghorse@gmail.com> Date: Tue Nov 2 12:11:42 2021 -0400 [BEAM-13001] fixes nil reference error in Extractor.ExtractFrom (#15865) commit 26545025233f3cbf9430350a432e2bedeaeaad50 Merge: 59bd9a0fc9 0bb6c1494f Author: Alexey Romanenko <33895511+aromanenko-dev@users.noreply.github.com> Date: Tue Nov 2 14:57:10 2021 +0100 Merge pull request #15811: [BEAM-12070] Make ParquetIO splittable by default commit 0bb6c1494fa4cfbb4dfce8a27bc575adb4be1b4d Author: Alexey Romanenko <aromanenko.dev@gmail.com> Date: Fri Oct 29 15:17:29 2021 +0200 [BEAM-12070] ParquetIO: use splittable reading by default commit 8830745867f7c86d18c24595eb75cf89afe15d20 Author: Dmitrii_Kuzin <dmitrii_kuzin@epam.com> Date: Tue Nov 2 12:03:26 2021 +0300 [BEAM-12730] Python. Custom delimiter add corner case commit 59bd9a0fc92fd4b9b8bd611a6a3845dc44fec321 Merge: 5103b8f760 59f57728a8 Author: Pablo <pabloem@users.noreply.github.com> Date: Mon Nov 1 20:32:20 2021 -0700 Merge pull request #15853 from [BEAM-13145][Playground] Validation error [BEAM-13145][Playground] Validation error commit 5103b8f76030e26c491b145a92bacf2e04897c13 Author: Ilya <ilya.kozyrev@akvelon.com> Date: Tue Nov 2 03:43:07 2021 +0300 Merge pull request #15840 from [Playground][BEAM-13146][Bugfix] playgroundPreCommit work incorrectly for frontend if flutter dependencies isn't installed * fix order for analyze task, starts after pubGet * add dependsOn instead of mustRunAfter Co-authored-by: damondouglas commit eb297f4883b422f4f56572c6b433e4f14b95e810 Merge: a17cd065ba 7c6c43f2e2 Author: Pablo <pabloem@users.noreply.github.com> Date: Mon Nov 1 17:41:23 2021 -0700 Merge pull request #15855 from [BEAM-13128] [Playground] Cleaning up environment variables in environment_service_test [BEAM-13128] [Playground] Cleaning up environment variables in environment_service_test commit 66ef31671d3cfbd19147efabf6ec97e606b4239a Author: Pablo Estrada <pabloem@apache.org> Date: Mon Nov 1 17:17:26 2021 -0700 JdbcIO has a single WriteFn underlying all implementations commit a17cd065bae25fb6742e60c4b0419b7262405157 Author: Ritesh Ghorse <riteshghorse@gmail.com> Date: Mon Nov 1 19:38:14 2021 -0400 [BEAM-13001] fixes Golint issues (#15859) commit e4f8f4b58b2b07be179cb5ca47a9281f01380861 Author: Luke Cwik <lcwik@google.com> Date: Fri Oct 29 17:42:37 2021 -0700 [BEAM-3811] Code clean-up of the CancelleableQueue to not throw InterruptedException on cancel/reset since we are never stuck in a loop here or waiting on a signal. Also update one location simplifying the queue interaction semantics to have the StreamObserver only interacted from the queue draining thread instead of relying on synchronization. commit a4e112974d8f3f51e646d9e714a4f8d456d69b4d Author: Benjamin Gonzalez <74670721+benWize@users.noreply.github.com> Date: Mon Nov 1 10:53:24 2021 -0600 [BEAM-11936] Fix errorprone warnings (#15821) * [BEAM-11936] Fix errorprone warnings: AssignmentToMock, AnnotateFormatMethod, BadImport * [BEAM-11936] Fix spotlessApply, fix errorprone warnings * [BEAM-11936] Fix dependency errors, add supresswarning * [BEAM-11936] Fix dependency errors, add supresswarning * [BEAM-11936] Fix dependency errors, add supresswarning * [BEAM-11936] Fix errorprone warning JodaPlusMinusLong * [BEAM-11936] Fix errorprone warning: CompareToZero * [BEAM-11936] Fix errorprone warning: SameNameButDifferent * [BEAM-11936] Fix errorprone warning: UnnecessaryAnonymousClass * [BEAM-11936] Fix spotlessApply, remove leftover code * [BEAM-11936] Fix errorprone warning: SameNameButDifferent supressed commit d5cd15ab4debbef0d967226bcebf2facc4d3e071 Author: Michael Luckey <25622840+adude3141@users.noreply.github.com> Date: Mon Nov 1 17:03:12 2021 +0100 [BEAM-13157] support hadoop configuration on ParquetIO.Parse commit 7c6c43f2e2b0ebd899c9cd1f237b1498bd88bcc9 Author: Pavel Avilov <avilovpavel6@gmail.com> Date: Mon Nov 1 18:16:22 2021 +0300 Added cleanup os env before and after tests commit 59f57728a8ba5f1490d2f4f6aa35bb0c317240c6 Author: AydarZaynutdinov <aydar.zaynutdinov@akvelon.com> Date: Mon Nov 1 16:27:31 2021 +0300 [BEAM-13145][Playground] add a new status: STATUS_VALIDATING; add a new status: STATUS_VALIDATION_ERROR; add a new status: STATUS_COMPILING; update of validation processing; commit b97a1c3698728f4389d6fa6c35c7b3768001567a Merge: 2c3e9799f7 08e0998e12 Author: Aydar Farrakhov <stranniknm@gmail.com> Date: Mon Nov 1 10:13:31 2021 +0300 Merge branch 'master' into BEAM-13105 # Conflicts: # playground/frontend/lib/modules/actions/components/new_example_action.dart commit 08e0998e12dda1b7754b2b05fcdcca5038295463 Author: Iñigo San Jose Visiers <42574310+InigoSJ@users.noreply.github.com> Date: Sat Oct 30 22:09:09 2021 +0100 Merge pull request #15753 from [BEAM-13080] Add option in Reshuffle to set num of keys generated. * [BEAM-13080] Add option in Reshuffle to set num of keys generated. * [BEAM-13080] Add option in Reshuffle to set num of keys generated. * [BEAM-13080] Add option in Reshuffle to set num of keys generated. * [BEAM-13080] Add option in Reshuffle to set num of keys generated. commit a5a0bd26ded0117240b2f6a967eb9f4c65209e6c Merge: cb9fb5110a 3fe3e1545f Author: Lukasz Cwik <lukecwik@gmail.com> Date: Fri Oct 29 14:25:12 2021 -0700 [BEAM-13015] Migrate bundle processing in the SDK harness to using BeamFnDataInboundObserver2 and BeamFnDataGrpcMultiplexer2. commit cb9fb5110aea50c3cbc2d7b49191b32d38a0307b Merge: d8cecb1475 ab2ddaa1bf Author: Lukasz Cwik <lukecwik@gmail.com> Date: Fri Oct 29 14:14:08 2021 -0700 [BEAM-13099] Add build files for vendored Calcite 1.28.0 commit d8cecb14750dc0baf475bee40a591cf1b1f5b1cb Author: Ritesh Ghorse <riteshghorse@gmail.com> Date: Fri Oct 29 15:54:58 2021 -0400 [BEAM-13001] DoFn metrics for Go SDK (#15657) commit 8cdc5274078f3242105801be43d6a202082a1054 Author: Andrew Pilloud <apilloud@google.com> Date: Fri Oct 29 11:25:01 2021 -0700 [BEAM-13118] Use branch for nightly "tag", tags are immutable commit 2e448dee58f1ee60551cc47b9aa7df6bc832734a Author: Chamikara Jayalath <chamikara@apache.org> Date: Fri Oct 29 09:59:34 2021 -0700 [BEAM-12047] Updates CHANGES.md to mention the URN convention (#15845) * Updates CHANGES.md to mention the URN convention * Updated text commit f048874b6d7b224058a712e6a21b1e5151c3bc39 Author: Brian Hulette <bhulette@google.com> Date: Fri Oct 29 09:32:09 2021 -0700 Minor: Remove broken python compatibility checks (#15828) commit 5ce06db5cdc402c8af4acdd9c9cbebfaa65f4d88 Author: Brian Hulette <bhulette@google.com> Date: Fri Oct 29 09:00:55 2021 -0700 [BEAM-13099] Use BlockBuilder.add(..) rather than BlockBuilder.append(Expressions.block(..)) (#15837) commit ab2ddaa1bf5804f35dfd01ae6712f0388209afc5 Author: Brian Hulette <bhulette@google.com> Date: Thu Oct 21 14:04:04 2021 -0700 [BEAM-13099] Modifications for vendor/calcite-1_28_0 build commit 8382982972dc59432778e85a4e6ec5808009e637 Author: Brian Hulette <bhulette@google.com> Date: Fri Oct 29 08:47:07 2021 -0700 [BEAM-13099] Copy vendored calcite build for 1.28.0 commit 67ef9aca13c274fab4179e1a553c77e2afc6d683 Author: Moritz Mack <mmack@talend.com> Date: Fri Oct 29 17:17:59 2021 +0200 [BEAM-8958] Use AWS credentials provider with BasicKinesisProvider (kinesis v1) (#15816) * [BEAM-8958] Use AWS credentials provider with BasicKinesisProvider (kinesis module using SDK v1) * fix comment * Address nullability issues * Commit missing test for AwsModule * Format AwsModuleTest commit f829123af1c26e2ebf9d33af580c64d909ec94c6 Author: mosche <mmack@talend.com> Date: Fri Oct 29 13:44:51 2021 +0200 Use vendored guava deps commit aa67b0f8b5b90da86357b129aedfecc7c0e5df73 Author: mosche <mmack@talend.com> Date: Fri Oct 29 12:41:42 2021 +0200 Apply format commit b60f73215c2c68c5db14c034008b50a1bd064f5c Author: mosche <mmack@talend.com> Date: Fri Oct 29 12:29:11 2021 +0200 Apply format commit 0831b758042c4c86bef28e54710a9bebec9287a0 Author: mosche <mmack@talend.com> Date: Fri Oct 29 12:16:09 2021 +0200 Bump localstack container version for DynamoDBIOTest to version used in KinesisIOIT commit aa5f06f7029b0bf0615893c76f80145ac943c608 Author: mosche <mmack@talend.com> Date: Fri Oct 29 12:15:05 2021 +0200 Bump localstack container version for more robust version with respect to HTTP/HTTPS and simplify KinesisIOIT in kinesis module. commit ec99053d66bb764f9bd67236c9f4651ad4a89add Author: mosche <mmack@talend.com> Date: Fri Oct 29 12:13:10 2021 +0200 [BEAM-11440] Add integration test for KinesisIO using Localstack on javaPostCommit commit 643825bc956df0e8fa9e6d10b2f10f317f4afb55 Merge: d07c52809c 8e0e23029e Author: Pablo <pabloem@users.noreply.github.com> Date: Thu Oct 28 21:59:25 2021 -0700 Merge pull request #15801 from [BEAM-12970][Playground] Implement gRPC API - GetCompileOutput [BEAM-12970][Playground] Implement gRPC API - GetCompileOutput commit d07c52809c5278ad0931555c42203976e3294cdc Author: Aydar Zainutdinov <aydar.zaynutdinov@akvelon.com> Date: Fri Oct 29 07:59:10 2021 +0300 Merge pull request #15800 from [BEAM-12970][Playground] Implement gRPC API - CheckStatus implement CheckStatus API method commit ce32c82740bf8a395feb90498338ddbd61c7c120 Author: Artur Khanin <artur.khanin@akvelon.com> Date: Fri Oct 29 07:58:18 2021 +0300 Merge pull request #15802 from [BEAM-12970][Playground] Implement gRPC API - GetRunOutput * Implemented GetRunOutput and covered with unit tests * Replaced logging library regarding PR comment commit 06d22077c904400502e3257a54fb6bfbd620dc33 Merge: 837e11c5b7 fd2d445ed4 Author: Pablo <pabloem@users.noreply.github.com> Date: Thu Oct 28 21:52:50 2021 -0700 Merge pull request #15824 from [BEAM-13120][Playground] GetSdkEnvsFromOsEnvs should check APP_WORK_DIR env before extracting [BEAM-13120][Playground] GetSdkEnvsFromOsEnvs should check APP_WORK_DIR env before extracting commit 837e11c5b75a46269e5562cd711bbf17757c1da9 Author: Aydar Zainutdinov <aydar.zaynutdinov@akvelon.com> Date: Fri Oct 29 07:47:00 2021 +0300 Merge pull request #15721 from [BEAM-13023][Playground] Implement Redis cache for pipelines' states * Implement local cache for pipelines' states * [BEAM-13005][Playground] add context; add redis option to NewCache; * [BEAM-13023][Playground] Add Redis implementation of Cache * [BEAM-13023][Playground] moved the code inside the methods from the individual methods; change RedisCache according to extends-like pattern; change grpclog to log * [BEAM-13023][Playground] add tests * [BEAM_13023][Playground] update go.sum * Edit GetValue method * [BEAM-13023][Playground] update go.sum * [BEAM-13023][Playground] add comments for cache.go; rename NewCache method to New * [BEAM-13023][Playground] remove constructor from cache.go rename constructor method from newLocalCache to New rename constructor method from newRedisCache to New move localCache files to new folder (local) move redisCache files to new folder (redis) * [BEAM-13023][Playground] Update regarding new place for generated files * [BEAM-13005][Playground] remove constructor from cache.go; rename constructor method from newLocalCache to New; move localCache files to new folder (local); add comments; * Resolve comments * [BEAM-13023][Playground] remove default expiration time; rename RedisCache to Cache * [BEAM-13023][Playground] update go.sum * [BEAM-13023][Playground] update go.sum * [BEAM-13023][Playground] add using logger Co-authored-by: Pavel Avilov <avilovpavel6@gmail.com> Co-authored-by: Ilya Kozyrev <ilya.kozyrev@akvelon.com> commit b06188653ccf2ac0315438ee0bc316e1cfa4500d Merge: 15682b40b6 552cca7ecc Author: Pablo <pabloem@users.noreply.github.com> Date: Thu Oct 28 21:04:19 2021 -0700 Merge pull request #15831 from [BEAM-13144] Handle runner-provided shards for TextIO [BEAM-13144] Handle runner-provided shards for TextIO commit 15682b40b6fabfea4128e7a9fc6ffdd34ba0e8a1 Merge: cd4b7f3b1a 2a6c80b4eb Author: tvalentyn <tvalentyn@users.noreply.github.com> Date: Thu Oct 28 20:04:51 2021 -0700 [BEAM-13143] Fix python doc generator error. #15830 commit 552cca7ecc05bd65585af748af8fefb689d148b8 Author: Pablo Estrada <pabloem@apache.org> Date: Thu Oct 28 15:38:44 2021 -0700 Handle runner-provided shards for TextIO commit 2a6c80b4eb69fc6e4fb0d9cf7e421c5b06172918 Author: Kyle Weaver <kcweaver@google.com> Date: Thu Oct 28 15:09:06 2021 -0700 [BEAM-13143] Fix python doc generator error. commit cd4b7f3b1af4f51bdab1a0b1a98f94b5288c09ec Merge: 4cdbe8dc3a 60eaa8256d Author: Brian Hulette <bhulette@google.com> Date: Thu Oct 28 11:06:07 2021 -0700 [BEAM-13099] Use RelMetadataQuery subclass rather than calling RelNode.metadata (#15820) [BEAM-13099] Use RelMetadataQuery subclass rather than calling RelNode.metadata commit 60eaa8256d2d6451d6defe1eaca783721c5a07f5 Author: Brian Hulette <bhulette@google.com> Date: Wed Oct 27 16:40:04 2021 -0700 [BEAM-13099] Update all rels to use BeamRelMetadataQuery commit 0e5451fc380365db9b738d22aae74e2a22d04154 Author: Brian Hulette <bhulette@google.com> Date: Wed Oct 27 16:39:41 2021 -0700 [BEAM-13099] Replace call to RelNode.metadata with BeamRelMetadataQuery commit 4cdbe8dc3a4be8fd3e6ee74cef43af3fb03761b1 Author: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Date: Thu Oct 28 13:14:23 2021 -0400 [BEAM-13130] Remove persistent references to stateKeyReaders (#15815) commit 290d9947af13477df6f08a9340f8eb30feac40a6 Merge: e8924960f3 af925d8520 Author: Alexey Romanenko <33895511+aromanenko-dev@users.noreply.github.com> Date: Thu Oct 28 19:13:41 2021 +0200 Merge pull request #15823: [BEAM-13138] Update code to retrieve container endpoint in DynamoDBIOTest (aws2) commit e8924960f3feb79ac7fd5403d1bfde4908598b7b Author: Miguel Hernandez <61989986+roger-mike@users.noreply.github.com> Date: Thu Oct 28 11:00:27 2021 -0500 Merge pull request #15294 from [BEAM-11986] Spanner write metric * [BEAM-11986] Spanner write metric * [BEAM-11986] Spanner read metric * [BEAM-11986] SpannerIO metrics integration tests * [BEAM-11986] SpannerIO metrics helper functions * [BEAM-11986] ServiceCallMetric instantiated before API call * [BEAM-11986] Skipped metrics tests on Dataflow commit d9d61311179a2a83ebcd5994d2a15fcdb5b93164 Merge: ec1decd21a ed8bba5639 Author: Pablo <pabloem@users.noreply.github.com> Date: Thu Oct 28 08:57:40 2021 -0700 Merge pull request #15822 from [BEAM-13126][Playground][Bugfix] Fix executor builder tests [BEAM-13126][Playground][Bugfix] Fix executor builder tests commit ec1decd21a3001a382e9b5ba0d220e62ac3b5ded Author: Alexandr Zhuravlev <stimbai@mail.ru> Date: Thu Oct 28 18:56:29 2021 +0300 Merge pull request #15709 from [BEAM-12967] [Playground] Create Example Selector initital structure * [BEAM-12967]: Created Examples Component * [BEAM-12967]: Wrapped CategoryBubble and ExpansionPanelItem in MouseRegion * [BEAM-12967] Added generated files * [BEAM-12967]: Added license to generated files * [BEAM-12967]: Deleted duplicate files * [BEAM-12967]: Formatted files commit fd2d445ed451be894901140bcf7e41ffb1cde125 Author: AydarZaynutdinov <aydar.zaynutdinov@akvelon.com> Date: Thu Oct 28 16:57:01 2021 +0300 [BEAM-13120][Playground] add working dir for setting up sdk environment; renaming; commit af925d8520ca0542755bd4553bad7f771f9148d3 Author: mosche <mmack@talend.com> Date: Thu Oct 28 15:18:53 2021 +0200 [BEAM-13138] Update code to retrieve container endpoint commit ed8bba563992b1e02a5e10279627a211200e0f6e Author: daria-malkova <daria.malkova@akvelon.com> Date: Thu Oct 28 12:44:18 2021 +0300 fix executor tests commit 8e0e23029e4cf405831091c334d368ece333a302 Author: Artur Khanin <artur.khanin@akvelon.com> Date: Thu Oct 28 10:59:21 2021 +0300 Replaced logging library regarding PR comment commit 02062eae4231ad25d55cfc158d0b3c4ffc2715d6 Author: Artur Khanin <artur.khanin@akvelon.com> Date: Tue Oct 26 16:29:09 2021 +0300 Implemented GetCompileOutput and covered with unit tests commit fbd0330679f50c0909bfa3b7c8db32dc4771e446 Merge: abf7ce29c3 9849940576 Author: tvalentyn <tvalentyn@users.noreply.github.com> Date: Wed Oct 27 23:27:57 2021 -0700 Fix Python docs build pre-commit failures. #15812 commit abf7ce29c35460f9a3f8920d09020f8416066c35 Merge: 7b0de72a84 fb990f28e7 Author: Pablo <pabloem@users.noreply.github.com> Date: Wed Oct 27 22:45:10 2021 -0700 Merge pull request #15803 from [BEAM-12970][Playground] Implement gRPC API - RunCode [BEAM-12970][Playground] Implement gRPC API - RunCode commit fb990f28e7c95300481bd6ccf42c089f8c9cb214 Author: AydarZaynutdinov <aydar.zaynutdinov@akvelon.com> Date: Thu Oct 28 08:29:53 2021 +0300 [BEAM-12970][Playground] add comments for tests; add TODO for finding java executable name; commit 7b0de72a84298f19365d3a2166822105d534c92b Author: Robert Bradshaw <robertwb@google.com> Date: Wed Oct 27 13:30:48 2021 -0700 [BEAM-13066] Disable using abstract iterable by default. (#15805) commit fa69c710ccc7195e5ffca7d7b5b50b5af27b22b4 Author: Robert Bradshaw <robertwb@google.com> Date: Wed Oct 27 13:30:02 2021 -0700 [BEAM-13066, BEAM-13087] Workaround for coder type combiner packing iterable issue on legacy dataflow runner (#15757) commit f8d3c1bed4a8b1f500fad61001cfa24d2da6afc3 Author: reuvenlax <relax@google.com> Date: Wed Oct 27 10:17:26 2021 -0700 Merge pull request #15731: [BEAM-13067] Mark GroupIntoBatches output as preserving keys commit 3fcc5843f5b2045f2e3caa3f761ff3d23daa0760 Merge: 86d429a803 e6fc61e6af Author: reuvenlax <relax@google.com> Date: Wed Oct 27 10:16:55 2021 -0700 Merge pull request #15796: [BEAM-13108] Support JSON types in Storage API sink commit 86d429a80306e5462c88250572673e3b83685664 Merge: 6220e476b2 5e3860acf7 Author: Pablo <pabloem@users.noreply.github.com> Date: Wed Oct 27 10:11:13 2021 -0700 Merge pull request #15808 from [BEAM-13118] Don't force push to master [BEAM-13118] Don't force push to master commit 2c3e9799f759f039fcc6c33be60036f72065cbc6 Author: Aydar Farrakhov <stranniknm@gmail.com> Date: Wed Oct 27 17:19:13 2021 +0300 [BEAM-13105] playground - add shorcut hint tooltip commit 4e9e44d920b36f8aa89277ea580112ed5a9f03eb Author: AydarZaynutdinov <aydar.zaynutdinov@akvelon.com> Date: Wed Oct 27 14:40:52 2021 +0300 [BEAM-12970][Playground] update processSuccess() method for compile step commit 6220e476b2801a69823d23aba3a53612da841d5f Merge: 0b9fb71b66 c4f0434aca Author: Alexey Romanenko <33895511+aromanenko-dev@users.noreply.github.com> Date: Wed Oct 27 13:09:08 2021 +0200 Merge pull request #15788: [BEAM-8958] Use AWS credentials provider with BasicKinesisProvider (AWS sdk v2) commit c4f0434acac303dee30c4f4a0105888cbfff2807 Author: mosche <mmack@talend.com> Date: Mon Oct 25 13:15:37 2021 +0200 [BEAM-8958] Use AWS credentials provider with BasicKinesisProvider (AWS sdk v2) rather than supporting basic credentials only. commit 98499405768126a5f1180ad4a814f95e1c3cba73 Author: Eugene Nikolaiev <eugene.nikolayev@gmail.com> Date: Wed Oct 27 12:41:43 2021 +0300 Fix Python docs build pre-commit failures. commit 59868135b7be8c2701d1a278c9641f9dbe29ae2f Author: AydarZaynutdinov <aydar.zaynutdinov@akvelon.com> Date: Wed Oct 27 11:08:34 2021 +0300 [BEAM-12970][Playground] simplify code; add using of logger; add documentation; commit 0b9fb71b66f3b2ecf55b680f522c9d2147656bfb Merge: 5cb634eb17 551c6d1010 Author: Heejong Lee <heejong@gmail.com> Date: Tue Oct 26 21:47:23 2021 -0700 Merge pull request #15606 from ihji/BEAM-12978 [BEAM-12978] Customizable dependency for Java external transform commit 5cb634eb173451c6dc3a2b328afdf985f3135366 Author: reuvenlax <relax@google.com> Date: Tue Oct 26 16:24:40 2021 -0700 [BEAM-13098] Fix translation of repeated TableRow fields (#15779) commit 63e21b11e0b8fca3839c6a8a1631d1d80a453245 Merge: 95d1f9ad2f cbce59fd0e Author: Lukasz Cwik <lukecwik@gmail.com> Date: Tue Oct 26 16:07:31 2021 -0700 [BEAM-11758] Update basics page: Splittable DoFn commit 5e3860acf7c168cfc8d7822eac3295544c3daf21 Author: Andrew Pilloud <apilloud@google.com> Date: Tue Oct 26 15:59:27 2021 -0700 [BEAM-13118] Don't force push to master commit 95d1f9ad2f59187220f35ffc86206a1e449cd122 Author: Lukasz Cwik <lukecwik@gmail.com> Date: Tue Oct 26 15:21:45 2021 -0700 [BEAM-4149] Ensure that we always provide and require the worker id. commit 3fe3e1545fbf31890b686199680fae909733de3e Author: Luke Cwik <lcwik@google.com> Date: Tue Oct 26 15:20:40 2021 -0700 [BEAM-13015] Migrate bundle processing in the SDK harness to using BeamFnDataInboundObserver2 and BeamFnDataGrpcMultiplexer2. commit cbce59fd0e79a11571a9999e8f65039c9e433e1b Author: Melissa Pashniak <melissapa@google.com> Date: Tue Oct 26 13:18:39 2021 -0700 Address review feedback commit 62b33a5bcd90ca1ef215e17b8e2d64d68c81615d Author: Tianyang Hu <htyleo@gmail.com> Date: Tue Oct 26 11:41:55 2021 -0700 Change sql.Options to an interface under sqlx. (#15790) The Options interface isn't supposed to be exposed to end users. This can also avoid potential circular dependencies. commit 84e24ea01f4ac7be3c3a8caeb735199bf0a80435 Merge: 46c649a254 e096daf974 Author: Pablo <pabloem@users.noreply.github.com> Date: Tue Oct 26 11:01:41 2021 -0700 Merge pull request #15500 from [BEAM-6721] Set numShards dynamically for TextIO.write() [BEAM-6721] Set numShards dynamically for TextIO.write() commit 46c649a2543f8f34e2cedf4702f70eb3cf1ab4f9 Merge: 49ae709e95 136eadc121 Author: Kyle Weaver <kcweaver@google.com> Date: Tue Oct 26 10:58:31 2021 -0700 Merge pull request #15789 from aromanenko-dev/BEAM-13104-ParquetIO-filter [BEAM-13104] ParquetIO: SplitReadFn must read the whole block commit 3bbcf5afbaa1da5c0878fc8535104adec3962051 Author: AydarZaynutdinov <aydar.zaynutdinov@akvelon.com> Date: Tue Oct 26 18:58:29 2021 +0300 [BEAM-12970][Playground] add setupCache() method; implement RunCode API method; change ExecutableName to a variable of LifeCycle instead of method; commit 49ae709e95c6f5fcd88dd4aba86398c111180279 Author: daria.malkova <daria.malkova@akvelon.com> Date: Tue Oct 26 10:21:54 2021 +0300 Merge pull request #15744 from [BEAM-13072][Playground] Executor builder * Add environment_service.go and structures for beam sdk, network envs and application envs * Add executor config structure to beam environment structure Fix tests, add new * add not yet supported * add java configuration file * Add executor builder * resolve conflicts with master * adapt server to new envs * Bugfix for a moved file * test bugfix Co-authored-by: Ilya Kozyrev <ilya.kozyrev@akvelon.com> commit e6fc61e6af053f7f1aab75fc26ebecf3ae853b4e Author: Reuven Lax <relax@google.com> Date: Mon Oct 25 21:27:22 2021 -0700 support json commit 0fb41701c22107e7938fa55be59b21b571ccd63c Merge: a9ac63b10c fd4eb15d65 Author: Melissa Pashniak <melissapa@google.com> Date: Mon Oct 25 17:37:39 2021 -0700 Merge branch 'master' into concepts4 commit fd4eb15d659e82bd370bd8d7cd7cb5f8fd174cbd Merge: e1b83baeb5 9cfcfbe7d4 Author: Lukasz Cwik <lukecwik@gmail.com> Date: Mon Oct 25 15:11:00 2021 -0700 [BEAM-12522] Enable side inputs on all splittable DoFn execution time related methods. commit e1b83baeb54d8fe1af2a200d7b5fde1c3c98ff3a Merge: b52932c34c 480e52cedb Author: Lukasz Cwik <lukecwik@gmail.com> Date: Mon Oct 25 13:53:59 2021 -0700 [BEAM-8543] Fix test filtering for Dataflow Runner V2 to exclude UsesStrictTimerOrdering commit 480e52cedb307948a9a7684f084250a938b7d3d3 Author: Luke Cwik <lcwik@google.com> Date: Mon Oct 25 13:08:57 2021 -0700 [BEAM-8543] Fix test filtering for Dataflow Runner V2 to exclude UsesStrictTimerOrdering The test names seem to have changed since they are not being excluded so swap to using test category exclusion. See https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2_Streaming/1378/testReport/ commit b52932c34cb5862068336711b4297a08bb9e47ef Author: Aydar Farrakhov <stranniknm@gmail.com> Date: Mon Oct 25 21:50:20 2021 +0300 Merge pull request #15783 from [BEAM-13048] [Playground] Add shortcuts * [BEAM-13048]: add keyboard shortcuts to playground * [BEAM-13048]: update shortcuts * [BEAM-13048]: fix lint errors commit 9b3198fdf592ed69d8dbbce1d65c48f52c14b1f6 Merge: 39613ea89f d463c0c359 Author: Pablo <pabloem@users.noreply.github.com> Date: Mon Oct 25 11:42:06 2021 -0700 Merge pull request #15787 from [BEAM-13062][Playground] Update errors in Protobuf [BEAM-13062][Playground] Update errors in Protobuf commit 39613ea89fee6ae888095fcb5ac9ca5fd6d687f9 Author: Melissa Pashniak <melissapa@google.com> Date: Mon Oct 25 11:26:28 2021 -0700 [BEAM-11758] Update basics page: Aggregation, Runner, UDF, Schema (#15763) * [BEAM-11758] Update basics page: Aggregation, Runner, UDF, Schema * Address most review feedback * Fix type in intro list * Address remaining two feedback comments commit b911642745e53015dd8b9dbcf8cb055f5713e35a Author: Aydar Zainutdinov <aydar.zaynutdinov@akvelon.com> Date: Mon Oct 25 21:25:27 2021 +0300 Merge pull request #15770 from [BEAM-13095][Playground] Using working directory from environment in LifeCycle instead of using hardcoded backend directory * [BEAM-13095][Playground] Change the logic of 'lc` so that it uses workingDir * [BEAM-13095][Playground] Update comments; Add using of filepath.Join() to construct filePath; * [BEAM-13095][Playground] Fix comment * [BEAM-13095][Playground] update go.sum commit 835d19ad8ef1756b306964f2f90b06b39165c2b5 Merge: 47a9b3fe45 aa26b53f63 Author: Pablo <pabloem@users.noreply.github.com> Date: Mon Oct 25 11:22:07 2021 -0700 Merge pull request #15654 from [BEAM-12998] [Playground] Environment service [BEAM-12998] [Playground] Environment service commit 47a9b3fe453fb6baf8e63b596ca4aaa8ed960814 Author: Pavel Avilov <avilovpavel6@gmail.com> Date: Mon Oct 25 21:21:13 2021 +0300 Merge pull request #15714 from [BEAM-13005] [Playground] Implement local cache for pipelines' states * Implement local cache for pipelines' states * [BEAM-13005][Playground] add context; add redis option to NewCache; * Edit GetValue method * [BEAM-13005][Playground] remove constructor from cache.go; rename constructor method from newLocalCache to New; move localCache files to new folder (local); add comments; * Resolve comments * [BEAM-13005][Playground] minor fix * add documentation for exported methods * fix documentation Co-authored-by: AydarZaynutdinov <aydar.zaynutdinov@akvelon.com> Co-authored-by: Ilya Kozyrev <ilya.kozyrev@akvelon.com> commit 43d3b90e5a066d1bb2fe80851c58e74162edc101 Author: Pavel Avilov <avilovpavel6@gmail.com> Date: Mon Oct 25 18:52:27 2021 +0300 Merge pull request #15772 from [BEAM-13032] [Playground] Implement the Log service * Implement the Log service * Resolve comments commit d463c0c35967e751892ca28e9c9d4f9ac4ec2c65 Author: AydarZaynutdinov <aydar.zaynutdinov@akvelon.com> Date: Mon Oct 25 18:37:31 2021 +0300 [BEAM-13062][Playground] update README file; generate files for backend commit aa26b53f633847947c535b48b574cd652a1a0c3f Author: Ilya Kozyrev <ilya.kozyrev@akvelon.com> Date: Mon Oct 25 18:33:34 2021 +0300 Add environment_service.go and structures for beam sdk, network envs, cache and application envs Co-authored-by: AydarZaynutdinov <aydar.zaynutdinov@akvelon.com> Co-authored-by: daria.malkova <daria.malkova@akvelon.com> commit 136eadc121e136e25aafc2b65f130526e7f20142 Author: Alexey Romanenko <aromanenko.dev@gmail.com> Date: Mon Oct 25 17:21:47 2021 +0200 [BEAM-13104] ParquetIO: SplitReadFn must read the whole block commit 6e821f7bb03438ef7141da6415ba330caea7bc31 Author: AydarZaynutdinov <aydar.zaynutdinov@akvelon.com> Date: Mon Oct 25 12:31:05 2021 +0300 [BEAM-13062][Playground] add STATUS_COMPILE_ERROR; add STATUS_RUN_TIMEOUT; update proto command in README file to generate files; commit 1f08d1f3ddc2e7bc7341be4b29bdafaec18de9cc Author: Pablo <pabloem@users.noreply.github.com> Date: Sun Oct 24 20:25:09 2021 -0700 Fixing BigQueryIO request too big corner case for streaming inserts (#15067) * Rewriting for readability * Address comments * Address comments * Addressing comments. Simplifying logic commit 551c6d1010335244c97e54ea13c2d552007bfc05 Author: Heejong Lee <heejong@gmail.com> Date: Thu Oct 21 21:01:12 2021 -0700 add ConfigT param to getDependencies method commit a9ac63b10c6d13e255e26424bc3e71c3bfd7aad7 Author: Melissa Pashniak <melissapa@google.com> Date: Thu Oct 21 19:06:01 2021 -0700 [BEAM-11758] Update basics page: Splittable DoFn commit 6173abb2e241865d89dff9ae679f4d422be84ee9 Author: Lukasz Cwik <lukecwik@gmail.com> Date: Thu Oct 21 18:40:10 2021 -0700 [BEAM-13015] Create a multiplexer that sends Elements based upon instruction id allowing for an inbound observer responsible for the entire instruction id. (#15747) * [BEAM-13015] Create a multiplexer that sends Elements based upon instruction id allowing for an inbound observer responsible for the entire instruction id. * fixup! checkstyle * Update sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexer2.java Co-authored-by: Brian Hulette <hulettbh@gmail.com> * fixup! Address PR comments. Co-authored-by: Brian Hulette <hulettbh@gmail.com> commit 3d8213a1bc7376e95cea37abaa3f7f05730f2581 Author: Calvin Leung <calvinleungyk@users.noreply.github.com> Date: Thu Oct 21 16:02:57 2021 -0700 [BEAM-11275] Support remote package download from remote filesystems in Stager (#15105) * [BEAM-11275] Updated CHANGES.md with new improvements * [BEAM-11275] Removed contextlib * [BEAM-11275] Added logic to download files from supported filesystems * [BEAM-11275] Update logs and catch larger scope Exception * [BEAM-11275] Formatting * [BEAM-11275] Download GCS file with FileSystems.open file handle * [BEAM-11275] Fixed file decompression issue * [BEAM-11275] Formatting * [BEAM-11275] Added tests for Stager._download_file * [BEAM-11275] Removed unused import * [BEAM-11275] Fix Python formatting commit ca6f4f08437fe2fa7c2319293dfc4574c60a6929 Author: Pablo <pabloem@users.noreply.github.com> Date: Thu Oct 21 14:48:46 2021 -0700 lazy creation of source splits for export-based ReadFromBigQuery (#15610) * lazy creation of source ssplits * formatting commit 8c143ab8dc2289c12fe4d7fc0b3042b0b518fda7 Merge: 975d71a5d6 30d1191580 Author: Daniel Oliveira <daniel.o.programmer@gmail.com> Date: Thu Oct 21 14:18:52 2021 -0700 Merge pull request #15777: [BEAM-12830] Avoiding read-only Go module cache in Gradle config. [BEAM-12830] Avoiding read-only Go module cache in Gradle config. commit 975d71a5d6311ed2ae33766581947de405fefabf Author: kileys <kileysok@google.com> Date: Thu Oct 21 13:56:36 2021 -0700 Test SetState addIfAbsent with no read (#15776) commit a4acdb8ad4fdbe48f49606a1f70fcb2416f1b40b Author: Melissa Pashniak <melissapa@google.com> Date: Thu Oct 21 13:22:10 2021 -0700 [BEAM-11758] Update basics page: Pipeline, PCollection, PTransform (#15720) * [BEAM-11758] Update basics page: Pipeline, PCollection, PTransform * Address review feedback * Minor PCollection text tweaks commit 406350249290ab7a702ed6c714db3fc84cda92aa Author: Heejong Lee <heejong@gmail.com> Date: Thu Oct 21 12:53:09 2021 -0700 rename variables commit 30d119158057b48069d91c4c79afabb5e30bd93d Author: Daniel Oliveira <daniel.o.programmer@gmail.com> Date: Thu Oct 21 12:30:59 2021 -0700 Avoiding read-only Go module cache in Gradle config. ./gradlew cleanTest was getting various errors from Go due to the Go module caches inside /.gogradle directories being read-only, so clean was unable to delete them. I adjusted this with the following: 1. Add -modcacherw to GOFLAGS on any commonly used gogradle tasks. 2. Avoid using gofmt on nested Gradle modules, as it would try to format the contents of /.gogradle. commit 76f531f42410999b8df3090afc203954f5d8ce1d Author: Chris Gray <50027088+chrismgrayftsinc@users.noreply.github.com> Date: Thu Oct 21 11:56:39 2021 -0700 [BEAM-13019] Add `containsInAnyOrder` with matchers to the `IterableAssert` interface (#15685) * [BEAM-13019] Add `containsInAnyOrder` with matchers to the `IterableAssert` interface * Fix the test. * Add a test * Fix a test * Fix another test. Confirmed all tests pass with `./gradlew compileTestJava` * Add a no-argument containsInAnyOrder to remove the ambiguity. * Update sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java * Update sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java Co-authored-by: Lukasz Cwik <lcwik@google.com> commit e6f7f4415610ff78eb669a6351795c5811cd98da Author: Robert Burke <lostluck@users.noreply.github.com> Date: Thu Oct 21 11:56:09 2021 -0700 [BEAM-13096] Double test timeout. (#15774) commit dc0548e20ca047cfa991d4138d63806e9d15086c Author: Damon Douglas <damondouglas@users.noreply.github.com> Date: Thu Oct 21 16:17:13 2021 +0000 Merge pull request #15761 from [BEAM-13008] Create gradle tasks for the Beam Playground * Implement Beam Playground gradle tasks * Fix trailing whitespace * Remove format from precommit * Fix order of includes and add descriptions * [BEAM-13008] add generated files and fix analysis issues (#2) * Reconcile errors with playgroundPrecommit output Co-authored-by: Aydar Farrakhov <stranniknm@gmail.com> commit 9c8939b8011f9003da77df1a10978211d23d3fde Author: Robert Burke <lostluck@users.noreply.github.com> Date: Wed Oct 20 19:16:38 2021 -0700 [BEAM-13082] Re-use dataWriter buffer. (#15762) commit 070ea22d507a2c26d838c872bb5790b8dad7f9b8 Author: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Date: Wed Oct 20 14:43:45 2021 -0400 [BEAM-11087] Add default WindowMappingFn from Main to Side Input windows, validation test, unit tests (#15743) commit 76535e701ec7fd917165bb6ff08883cf016f1e56 Merge: 59f41e59a4 c5b367c01e Author: Tim Robertson <timrobertson100@gmail.com> Date: Wed Oct 20 20:26:52 2021 +0200 Minor: corrections to flink runner docs (#15759) commit 59f41e59a4833c306333c2f2a10c2f730d53c9e7 Merge: a05aa45dac b8ed280b83 Author: Kenn Knowles <kenn@apache.org> Date: Wed Oct 20 11:18:01 2021 -0700 Merge pull request #15739: [BEAM-13073] Fixes Unexpected GC when using Java 11 commit c5b367c01e7a009bf142be2ad9a156b23f2f7fd0 Author: mosche <moritz@mackmail.net> Date: Wed Oct 20 17:59:05 2021 +0200 adhoc: Minor update to flink runner docs commit a05aa45dac5e7295a56634f0916b7019bc6782e5 Merge: 8578c9a935 f7691e691d Author: Etienne Chauchot <echauchot@apache.org> Date: Wed Oct 20 09:38:36 2021 +0200 Merge pull request #15381 from egalpin/BEAM-10990-elasticsearch-response-filtering Merge pull request #15381: [BEAM-10990] Elasticsearch response filtering and [BEAM-5172] Tries to reduce ES UTest flakiness commit 8578c9a9352a8a5c9621647e9c585321f9761dc5 Author: Chamikara Jayalath <chamikara@apache.org> Date: Tue Oct 19 20:03:08 2021 -0700 [BEAM-13079] Updates cross-language transform URNs to use the new convention (#15748) * Updates cross-language transform URNs to use the new convention * Fix the Spanner Insert URN. Co-authored-by: Brian Hulette <hulettbh@gmail.com> * Fix the Kafka test * Fixes the Pub/Sub test failure Co-authored-by: Brian Hulette <hulettbh@gmail.com> commit 4d1c2af0b22058a74e14aa3002efba7ebb8bb632 Merge: bfe5feabf9 90251e631b Author: Pablo <pabloem@users.noreply.github.com> Date: Tue Oct 19 16:23:00 2021 -0700 Merge pull request #15750 from [BEAM-13077][Playground] LifeCycle.CreateFolders() after several uses returns error [BEAM-13077][Playground] LifeCycle.CreateFolders() after several uses returns error commit bfe5feabf9a70aa250c1142b01f4379e1f124d6e Merge: b76c7ee041 8892810ad0 Author: Kyle Weaver <kcweaver@google.com> Date: Tue Oct 19 12:52:02 2021 -0700 Merge pull request #15755 from ibzib/depth1 [BEAM-13055] Use unshallow clone to create PR. commit 8892810ad04b9c16f59334be5c2954fe2204a0a2 Author: Kyle Weaver <kcweaver@google.com> Date: Tue Oct 19 11:10:12 2021 -0700 [BEAM-13055] Use unshallow clone to create PR. commit b76c7ee041b10941d5de18a083fa5ee4a6626e48 Merge: aa4edda39c 650b514618 Author: Pablo <pabloem@users.noreply.github.com> Date: Tue Oct 19 10:35:10 2021 -0700 Merge pull request #15656 from [BEAM-12983] [Playground] Initial gRPC server [BEAM-12983] [Playground] Initial gRPC server commit aa4edda39ceb8d7a80f56bd37caa6233dba7de5d Merge: a4dc6f3b8e 63d9c621e1 Author: Yichi Zhang <zyichi@google.com> Date: Tue Oct 19 08:58:26 2021 -0700 Merge pull request #15716 from scwhittle/fix_unexpected_blocking [BEAM-13042] Prevent unexpected blocking in RegisterAndProcessBundleOperation.hasFailed commit a4dc6f3b8ee64d9897e9c045f8b1acbbc008edd2 Author: Tianyang Hu <htyleo@gmail.com> Date: Tue Oct 19 08:49:38 2021 -0700 [BEAM-13068] Add a SQL API in Beam Go SDK (#15746) * Add a SQL API in Beam Go SDK * User-specified expansionAddr shouldn't be overridden. commit f7691e691d8600bc5b70539c07ed0eb843955b04 Author: egalpin <egalpin@users.noreply.github.com> Date: Wed Aug 25 09:30:42 2021 -0400 [BEAM-5172] Tries to reduce ES uTest flakiness commit 6bfd83d44aa27667773a8025a5a4bcb6a847a367 Author: egalpin <egalpin@users.noreply.github.com> Date: Fri Aug 13 14:16:01 2021 -0400 [BEAM-10990] Adds response filtering for ElasticsearchIO commit 90251e631beee35f1847025147d55227f3ba999d Author: AydarZaynutdinov <aydar.zaynutdinov@akvelon.com> Date: Tue Oct 19 14:01:57 2021 +0300 [BEAM_13077][Playground] fix logic for CreateFolders method commit 650b514618001f53755d62e52f17a5351501cd42 Author: Ilya Kozyrev <ilya.kozyrev@akvelon.com> Date: Tue Oct 19 12:40:51 2021 +0300 Implement initial server with grpcweb wrapper commit 70ac565ea71f247fbe545489a5916ed058f4ddac Merge: abe3aa768e d0253ed545 Author: Lukasz Cwik <lukecwik@gmail.com> Date: Mon Oct 18 16:26:01 2021 -0700 [BEAM-13015] Implement a simplified cancellable blocking queue with similar semantics to ArrayBlockingQueue. commit d0253ed54535e00c61d6c3e04d68f35bbf2d46f9 Author: Luke Cwik <lcwik@google.com> Date: Fri Oct 15 15:53:22 2021 -0700 [BEAM-13015] Implement a simplified cancellable blocking queue with similar semantics to ArrayBlockingQueue. There are a few places where we use a "POISON" object to signal the end of the queue. This implementation can more safely replace those. commit abe3aa768e1e3bd4ece9d1bf4cba31424b0ec341 Merge: ffde2a6020 9343f37cf7 Author: Kyle Weaver <kcweaver@google.com> Date: Mon Oct 18 13:24:04 2021 -0700 Merge pull request #15603 from zhoufek/fat [BEAM-9487] Various Trigger.may_lose_data fixes commit ffde2a60205d920749787a67d64a0f2af199ba35 Merge: debbd7121d a9ec3c8d94 Author: Robert Bradshaw <robertwb@google.com> Date: Mon Oct 18 11:54:35 2021 -0700 Merge pull request #15719 from [BEAM-13040] Revert breaking PR and add some test cases. commit debbd7121dcbdbbe44fd3e6b89db8c53568c18b2 Author: Tianyang Hu <htyleo@gmail.com> Date: Mon Oct 18 11:51:03 2021 -0700 [BEAM-13068] Add xlangx.DecodeStructPayload (#15741) commit 8e6d8de34beaf3afa4faf38703b1bc12a2212329 Merge: 85a122735f 279949d025 Author: Andrew Pilloud <apilloud@users.noreply.github.com> Date: Mon Oct 18 11:26:21 2021 -0700 Merge pull request #15728: [BEAM-12393] sql support for Zeta Sql commit 85a122735f84c0ee46ba0fb583d9ff9e05dcf2fc Author: Benjamin Gonzalez <74670721+benWize@users.noreply.github.com> Date: Mon Oct 18 13:11:10 2021 -0500 [BEAM-11936] Fix some errorprone warnings (#15648) * [BEAM-11936] Fix errorprone AutoValueFinalMethods warnings * [BEAM-11936] Fix errorprone warnings: AssignmentToMock, AnnotateFormatMethod, BadImport * [BEAM-11936] Fix errorprone warnings: JodaToSelf, MathAbsoluteRandom, BigDecimalLiteralDouble, JodaDurationConstructor * [BEAM-11936] Fix errorprone warnings: UnusedMethod, UnnecessaryParentheses * [BEAM-11936] Fix spotlessApply, fix errorprone warnings * [BEAM-11936] Fix dependency errors, add supresswarning * [BEAM-11936] Fix dependency errors, add supresswarning * [BEAM-11936] Fix dependency errors, add supresswarning * [BEAM-11936] Fix errorprone warnings: AnnotateFormatMethod, add errorprone dependency as compileOnly * [BEAM-11936] Add errorprone dependency to samza project as compileOnly * [BEAM-11936] Add FormatStringAnnotation, change errorprone dependencies to common dependencies commit bf91bd75ea0e66b5fe1f47b73ab85f382f550203 Merge: de396a6fd1 fb6d5546e7 Author: Pablo <pabloem@users.noreply.github.com> Date: Mon Oct 18 11:10:31 2021 -0700 Merge pull request #15645 from [BEAM-12988] [Playground] Add FileSystem service [BEAM-12988] [Playground] Add FileSystem service commit de396a6fd169a1dac20d6f97c9115b1ba12e3cd2 Author: Robert Bradshaw <robertwb@google.com> Date: Mon Oct 18 10:29:51 2021 -0700 [BEAM-13066] Produce abstract iterables from IterableCoder. (#15662) Currently this will non-deterministically produce lists, generally for small data, which makes it easy to write code that works in tests but not in production. commit b8ed280b83cd89053aabd1ec5548ae028f8d82c6 Author: Luís Bianchin <labianchin@users.noreply.github.com> Date: Mon Oct 18 14:27:16 2021 +0200 Add -XX:+AlwaysActAsServerClassMachine to J…

Adds ReadChangeStreamPartitionDoFn, which is an SDF to read partitions from change streams and process them accordingly. This component receives a change stream name, a partition, a start time and an end time to query. It then initiates a change stream query with the received parameters. Within a change stream, 3 types of records can be received: 1. A Data record 2. A Heartbeat record 3. A Child partitions record Upon receiving #1, the function updates the watermark with the record's commit timestamp and emits the record into the output PCollection. Upon receiving #2, the function updates the watermark with the record's timestamp, but it does not emit any record into the PCollection. Upon receiving #3, the function updates the watermark with the record's timestamp and writes the new child partitions into the metadata table. These partitions will be later scheduled by the DetectNewPartitions component. Once the change stream query for the element partition finishes, it marks the partition as finished in the metadata table and terminates.

Summary: test commit Merge branch 'apache:master' into master Merge branch 'apache:master' into master Update README with latest PreCommit Jobs Update Postcommit jobs with latest jobs Update Performace job tests in readme update load job tests with latest updates update other jobs test with latest updates mismatch links fix update trigger phrase for some postCommit jobs correct trigger phrases in readme [BEAM-12391] update avro sink to close the opened file handle, instead of getting it by reference from the writer. This ensures that when we load the python writer implementation we don't run into an attribute error [BEAM-13482] Python fully qualified name external transforms. Add Python expansion service entry point. [BEAM-13509] Stop sharing SQS client between readers of same source. [BEAM-13565][Playground] Add GetPreparationOutput method into the .proto files; Regenerate files; update bom [BEAM-13569] Change Spark dependencies to implementation. They are needed to spin up a standalone Spark cluster. remove redundant dependency python sdk examples: Fixed typo in wordcount example. [BEAM-13591] Bump log4j2 version to 2.17.1 [BEAM-13459] Update CHANGES.md, add note about artifact caching for python jobs [BEAM-13587] Attempt to load AWS region from default provider chain in AwsOptions Merge pull request #16405: [BEAM-13587] Attempt to load AWS region from default provider chain in AwsOptions Merge pull request #16368: [BEAM-13509] Stop sharing SQS client between readers of same source. Add Flink runner support for OrderedListState. This version reads the list entirely into memory, so will not work well for large lists. Merge pull request #16404: [BEAM-13586] Fix NPE in DefaultS3ClientBuilderFactory Fix sdk_container_builder too many values to unpack error [BEAM-13480] Sickbay PubSubIntegrationTest.test_streaming_data_only on Dataflow (#16255) remove redundant testImplementation dependencies Merge pull request #16417 from y1chi/python_val [BEAM-13459] Fix sdk_container_builder too many values to unpack error [BEAM-13430] Swap to use "mainClass" instead of "main" since it was deprecated. (#16400) See https://docs.gradle.org/7.3.2/dsl/org.gradle.api.tasks.JavaExec.html?_ga=2.124534154.1653102406.1640909413-346869268.1640099071#org.gradle.api.tasks.JavaExec:main for details Fix remaining failing perf IT tests. [BEAM-13430] Replace deprecated "appendix" with "archiveAppendix" (#16401) See https://docs.gradle.org/7.3.2/dsl/org.gradle.api.tasks.bundling.AbstractArchiveTask.html?_ga=2.44668452.1653102406.1640909413-346869268.1640099071#org.gradle.api.tasks.bundling.AbstractArchiveTask:appendix for more details. Merge pull request #16415 from deadwind4/BEAM-13591 [BEAM-13015] Add jamm as a java agent to the Java SDK harness container (#16412) This allows for accurate object sizing for caching. [BEAM-13430] Partially revert https://github.com/apache/beam/commit/eaacf709a861ea1c260e3856b720266348b1aecc (#16419) Bump the version back to 2.37 Merge pull request #16246: [BEAM-12391] update avro sink to close the opened file handle directly Merge pull request #16266 [BEAM-13482] Python fully qualified name external transforms. [BEAM-13482] Python fully qualified name external transforms. Merge pull request #15863 from [BEAM-13184] Autosharding for JdbcIO.write* transforms * Supporting autosharding on JdbcIO.write transforms * Making autosharding optional * Adding validation * integration test * Reducing code duplication * Adding a maximum bundle size to avoid overwhelming the memory Merge pull request #15984: [BEAM-2791] OrderedListState for Flink [BEAM-11936] Enable FloatingPointAssertionWithinEpsilon errorprone check (#16261) [BEAM-11936] Enable LockNotBeforeTry errorprone check (#16259) [BEAM-11936] Enable errorprone unused checks (#16262) Add Nexmark Query 14 (#16337) This change adds Query14.java for testing the performance of Reshuffle transform. Co-authored-by: Arun Pandian <pandiana@google.com> Merge pull request #16396 from ibzib/BEAM-13569 [BEAM-13569] Change Spark dependencies to implementation. [BEAM-13015] Migrate all user state and side implementations to support caching. (#16263) This change also ensures that prefetch can be invoked on the iterable to prevent the prefetch being lost once the iterator is discarded. See https://s.apache.org/beam-fn-state-api-and-bundle-processing#heading=h.tms0ncgbzz6f Merge pull request #16418 from y1chi/perf-it-2 Fix remaining failing perf IT tests. [BEAM-13459] Update CHANGES.md, add note about artifact caching python jobs #16416 Upgrade python library versions in base_image_requirements.txt [BEAM-13015] Use 20% of memory when the maximum has been configured. (#16420) * [BEAM-13015] Use 20% of memory when the maximum has been configured. The boot.go always sets the -Xmx value but we need a fallback incase used in different environments. [BEAM-13567] Consolidate runner flag definition. (#16426) Merge pull request #16424 from y1chi/bump_pubsub Update python library versions in base_image_requirements.txt [BEAM-13601] Don't cache Row types for a schema. (#16427) This fixes apache_beam.tools.microbenchmarks_test.MicrobenchmarksTest.test_coders_microbenchmark. Also separate out the named tuple and row cases as they have little in common at this point. [BEAM-13430] Re-enable checkerframework (#16429) This fixes a bad merge conflicts that occurred on https://github.com/apache/beam/pull/16395 [BEAM-13430] Ensure that testRuntimeMigration depends on "default" configuration allowing us to get the main "jar" as a dependency. (#16430) This better mirrors what the "shadow" and "shadowTest" configurations do and what "testRuntime" used to do with gradle 6.x and earlier. Merge pull request #16277 from [BEAM-13124][Playground] Create readiness endpoint * [BEAM-13124][Playground] Implement handle func for /readiness address; Add deleting all prepared files/folders for RunCode Api method in case of any error with the cache; * [BEAM-13124][Playground] renaming * [BEAM-13124][Playground] refactoring; add check for numOfParallelJobs's value; * [BEAM-13124][Playground] merge with master; fix staticcheck issues; * [BEAM-13124][Playground] rename tests * [BEAM-13124][Playground] update comments Merge pull request #16314 from [BEAM-13260][Playground]Implement setup of CI test environment * Add module to create GKE cluster via terraform, helm chart to deploy to GKE and run examples CI and GH action to run all that * Update comments sections Merge pull request #16383 from [BEAM-13566][Playground] Add logic of sending preparation's output on the backend side * [BEAM-13565][Playground] Add GetPreparationOutput method into the .proto files; Regenerate files; * [BEAM-13566][Playground] Implement logic to save prepare step output Merge pull request #16382 from [BEAM-13565][Playground] Add GetPreparationOutput API method to .proto file [BEAM-13565][Playground] Add GetPreparationOutput API method to .proto file Merge pull request #16365 from [BEAM-13559][Playground] Remove tag in examples CD * [BEAM-13559][Playground] Remove beam playground tag from examples for CI/CD steps; Fix tests; * [BEAM-13559][Playground] Update test Merge pull request #16360 from [BEAM-13546][Playground] Update nginx configuration to enable embedding iframes * [Playground][BEAM-12941][Bugfix] Fix workflows for playground applications (#83) * Update workflows for playground * Attempt to fix tests * Remove continue on error to catch errors * Fix linter problem for backend dockerfile * Update folder to run backend go linter * Moved flutter test to execution via gradle tasks * Revert "[Playground][BEAM-12941][Bugfix] Fix workflows for playground applications (#83)" (#88) This reverts commit b73f5f70ac4184b56a0a03922731c5f2f69b9566. * Added support for embedded iframe and also gzip compression of assets * Add gzip_static config to nginx Co-authored-by: Ilya <ilya.kozyrev@akvelon.com> Co-authored-by: Aydar Zainutdinov <aydar.zaynutdinov@akvelon.com> Co-authored-by: daria.malkova <daria.malkova@akvelon.com> Co-authored-by: Pavel Avilov <pavel.avilov@akvelon.com> Co-authored-by: Aydar Farrakhov <stranniknm@gmail.com> Merge pull request #16192 from [BEAM-13395] [Playground] Tag katas * Tag katas * Update tags * Update katas * Update playground/categories.yaml Co-authored-by: Ilya <ilya.kozyrev@akvelon.com> * Remove the repeated lines in categories.yaml; Edit tag; Co-authored-by: daria.malkova <daria.malkova@akvelon.com> Co-authored-by: Ilya <ilya.kozyrev@akvelon.com> Merge pull request #16254 from [BEAM-13249][Playground] Security – Mock Network * Added proxy server with allow-list using mitmproxy * Added updated mock network dockerfile * Updated the format of proxy env variables; added proxy for Python backend * Added proxy for Go backend * Add python certificat * Stylefixes * Remove redundant Dockerfile * Added licenses * Added GCS to allow-list * Add allowed buckets * Applied Python style and changed comments * change name of lists * change name of lists * change name of the public bucket Co-authored-by: Sergey Kalinin <sergey.kalinin@akvelon.com> Co-authored-by: daria-malkova <daria.malkova@akvelon.com> Merge pull request #16377: [BEAM-13576] update bom [BEAM-12879] Prevented missing permission from failing GCS I/O 1. Errors caused by metrics gathering no longer fail the GCS I/O. 2. Added a debug level log about potential permission issue and missing project id label for reported metrics. 3. Added a test for the failing execution route. Merge pull request #16347: fix: move connector to use v1 BigQuery Storage Write API [BEAM-12879] Prevented missing permission from failing GCS I/O [BEAM-13603] Fix bug in apache_beam.utils.Shared (#16437) Co-authored-by: Ahmet Altay <aaltay@gmail.com> [BEAM-10345] Add an import guard to support recent google-cloud-spanner versions. (#16434) * Add an import guard to support recent google-cloud-python versions. * Ignore when library is not available [BEAM-13091] Generate missing staged names from hash for Dataflow runner add test [BEAM-13604] NPE while getting null from BigDecimal column Fixed empty labels treated as wildcard when matching cache files (#16440) 1. Fixed a test watching nested dictionaries by watching the concerned PCollections directly; 2. Fixed cache manager's "exists" implementations to avoid treating empty labels as wildcard when matching cache files. This bug is not a security issue but could cause unexpected behavior when getting materialized values of PCollections that are not cached. [BEAM-13570] Remove erroneous compileClasspath dependency. (#16438) * [BEAM-13570] Remove erroneous compileClasspath dependency. * Exclude unsupported test categories. [BEAM-13015] Plumb through process wide and bundle cache through the FnApiStateAccessor. (#16423) [BEAM-13015] Cache the state backed iterable used for large GBK results. (#16409) Merge pull request #16442: [BEAM-13604] NPE while getting null from BigDecimal column Fix formatting/alignment (#16443) Merge pull request #16183 from [BEAM-13427] [Playground] show logs for precompiled objects * [BEAM-13411][Playground] Add method to get logs of precompiledObjects * [BEAM-13427] show logs for precompiled objects * [BEAM-13427] add support for dots in pipeline optons * [BEAM-13427]: revert go file * [BEAM-13427]: fix build * [BEAM-13427] playground refactoring * [BEAM-13427] playground print logs errors Co-authored-by: AydarZaynutdinov <aydar.zaynutdinov@akvelon.com> [BEAM-10277] re-write encoding position tests to declare schema protos explicitly (#16267) Update local_env_tests.yml (#16444) [BEAM-13574] Filesystem abstraction Rename support (#16428) [BEAM-13597] Setup Go in github actions (#16446) [BEAM-13602] Prevented metrics gathering from failing bigtable io 1. bigtable mutate row might return None instead of Status when running into retryable errors. Handled the special case as DEADLINE_EXCEEDED because bigtable must have exhausted the retry timeout or when there is no retry policy inplace. 2. For all rows mutated, if the response status is not ok, that data might have lost. Added a TODO item for that. Merge pull request #15765 from ihji/BEAM-13091 [BEAM-13091] Generate missing staged names from hash for Dataflow runner Merge pull request #16161 from [BEAM-12164] Add Spanner Partition Metadata DAOs * [BEAM-12164] Add Spanner Partition Metadata DAOs * fix: remove metrics table from DAO * fix: fix compilation error * chore: fix linting violations * feat: add opencensus dependency * deps: update OpenCensus API to 0.30.0 Merge pull request #16203 from [BEAM-12164] Add Spanner Change Stream Mappers * [BEAM-12164] Add Spanner Partition Metadata DAOs * fix: remove metrics table from DAO * fix: fix compilation error * chore: fix linting violations * feat: add opencensus dependency * [BEAM-12164] Add Spanner Change Stream Mappers The mapper classes convert from Cloud Spanner Structs to the change stream models used by the connector. There are two mappers implemented: 1. For mapping to partition metadata models. 2. For mapping to change stream records (one of heartbeat, data or child partitions). * deps: update OpenCensus API to 0.30.0 [adhoc] Fix BigTableIO description Merge pull request #16453: [adhoc] Fix BigTableIO description [BEAM-8727] Bump software.amazon.awssdk to 2.17.106 make the code more pythonic [BEAM-13015] Remove dead code now that all instances have migrated to the state caching implementation. (#16447) [BEAM-13386] Add RLock support for cloudpickle (#16250) Fix overflow Merge pull request #16433 from danthev/py_overflow_fix [BEAM-13599] Fix overflow error in Python Datastore RampupThrottlingFn don't close streams in finalize Loosen typing extensions bound Merge pull request #16456: [BEAM-8727] Bump software.amazon.awssdk to 2.17.106 [BEAM-4868] Bump com.amazonaws to 1.12.135 [BEAM-13243][BEAM-8374] Add support for missing PublishResponse fields in SnsIO.Write (AWS SDK v2) Merge pull request #16454: [BEAM-4868] Bump com.amazonaws to 1.12.135 [BEAM-12092] Bump jedis to version 4.0.1 (#16287) Merge pull request #16464: [BEAM-13617] don't close streams in finalize [BEAM-13534] Add automated port polling to expansion service runner if port isn't provided (#16470) Merge pull request #16344 from [BEAM-13536][Playground][Bugfix] CI step doesn't log case with empty category value * [BEAM-13536][Playground] Add processing of empty value in Beam tag validation method * [BEAM-13536][Playground] Add comments Merge pull request #16359 from [BEAM-13545][Playground] Add GetValidationOutput API method to .proto file * [BEAM-13545][Playground] Add GetValidationOutput method into the .proto files; Regenerate files; * [BEAM-13545][Playground] Regenerating proto files Merge pull request #16384 from [BEAM-13308] [Playground] Getting baseFileFolder from environment * Getting baseFileFolder from environment * Replace 2 args to filepath.join on some methods * Refactoring code Merge pull request #16306 from [BEAM-13447] [Playground] Add filling of the chosen thumbs up/down button * [BEAM-13428] add google analytics to the playground * [BEAM-13428]: print analytics error on console * BEAM-13252] Feedback Thumbs Up/Down Functionality * [BEAM-13252] Deleted unnecessary comment * [BEAM-13447]: playground - add filling of the chosen thumbs up/down button * [BEAM-13447] playground - dont save feedback state on persitent storage * [BEAM-13447] update pubspec Co-authored-by: Alexander <alexander.zhuravlev@akvelon.com> Merge pull request #16361 from [BEAM-13543][Playground] Add logic of sending validation's output on the backend side * [BEAM-13545][Playground] Add GetValidationOutput method into the .proto files; Regenerate files; * [BEAM-13543][Playground] Implement logic to save validation output * [BEAM-13543][Playground] minor refactoring * [BEAM-13543][Playground] change back from string to []byte * [BEAM-13545][Playground] Regenerating proto files [BEAM-12562] Dataframe pipe implementation (#16256) * [BEAM-12565] Series implementation of compare * [BEAM-12565] DataFrame implementation of compare * [BEAM-12565] Fixed minor issues and error checks * [BEAM-12562] Initial pipe implementation for DataFrame * [BEAM-12562] Pipe implementation for Series * [BEAM-12562] Pipe implementation for GroupBy * [BEAM-12562] Added tests for Series and GroupBy pipe * [BEAM-12562] Support for (callable, data_keyword) func in DataFrame, Series, and GroupBy Co-authored-by: Brian Hulette <bhulette@google.com> Merge pull request #16338 from [BEAM-13528][Playground] Add liveness check * [BEAM-13528][Playground] add liveness endpoint * [BEAM-13528][Playground] update comment [BEAM-13626] Remap expanded outputs after merging. (#16471) Merge pull request #16147 from [BEAM-13359] [Playground] Tag existing examples * Tag examples * Refactoring tags * Add multifile to tag * Updated categories of some examples * Edit tags * Edit tags * Update tags * fix spotless check * Update tags * MinimalWordCount.java and DistinctExample.java reads only one file * Fix checks * Change examples which read * from bucket to read one file * Update tags; Co-authored-by: Artur Khanin <artur.khanin@akvelon.com> Co-authored-by: daria-malkova <daria.malkova@akvelon.com> Co-authored-by: Pavel Avilov <pavel.avilov> [BEAM-3221] Improve documentation in model pipeline protos (#16474) * clarify state caching tokens * fix documentation around side input types and valid request types. [BEAM-13614] Add OnWindowExpiration support to the Java SDK harness and proto translation. (#16458) This implementation adds a timer family spec in the event time domain and adds the field to the ParDoPayload mentioning which timer family spec represents the on window expiration callback. Optional args and kwargs for named external transforms. Merge pull request #16156 from [BEAM-13391] Fix temporary file format in WriteToBigQuery * Fix temporary file format in WriteToBigQuery * Change a desription * Fix pylint issue * Import BigQueryDisposition class * Combine both tets together * Fix lint issues Co-authored-by: Sayat Satybaldiyev <sayat.satybaldiyev@getcruise.com> Co-authored-by: Pablo <pabloem@users.noreply.github.com> Loosen typing extensions bound #16466 [adhoc] Run spotlessApply on java examples to fix master Merge pull request #16479: [adhoc] Run spotlessApply on java examples to fix master [BEAM-8806] Integration test for SqsIO [BEAM-13631] Add deterministic SQS message coder to fix reading from SQS in batch mode Merge pull request #16128: [BEAM-13243][BEAM-8374] Add support for missing PublishResponse fields in SnsIO.Write (AWS SDK v2) [BEAM-13628] Update SideInputCache to use full Transform and SideInputIDs as token information (#16483) [BEAM-13602] Prevented metrics gathering from failing bigtable io Merge pull request #16475 Optional args and kwargs for named external transforms. [BEAM-13432] Skip ExpansionService creation in Job Server (#16222) * [BEAM-13432] Update Java ExpansionService to be configurable of Environment 1. Update ExpansionService to use its own option instead of blindly inherit from pipeline when registering environment so it can be configurable. 2. Update JobServerDriver to skip creating expansion server when expansion port is negative. * Revert ExpansionService change * Add unit tests to test job server start up without expansion service. * Apply spotless check [BEAM-13616] Initial files for vendored gRPC 1.43.2 (#16460) Co-authored-by: Lukasz Cwik <lcwik@google.com> [BEAM-13638] Datatype of timestamp fields in SqsMessage for AWS IOs for SDK v2 was changed from String to long, visibility of all fields was fixed from package private to public Merge pull request #16491: [BEAM-13638] Fix visibility of fields in SqsMessage (AWS Sdk v2) [BEAM-13641][Playground] Add SCIO SDK support on the CI/CD step Merge pull request #16469 from [BEAM-13623][Playground] [Bugfix] During unit tests failing there is no any output * [BEAM-13623][Playground] Add getting of error's output from `RunOutput` in case unit test is failed * [BEAM-13623][Playground] fix of tests Merge pull request #16149 from [BEAM-13113] [Playground] playground frontend documentation * [BEAM-13113] playground frontend documentation * [BEAM-13113] playground frontend documentation structure * [BEAM-13113] fix flutter command * [BEAM-13113] remove trailing spaces * [BEAM-13133] update beam playground frontend readme Co-authored-by: Artur Khanin <artur.khanin@akvelon.com> * [BEAM-13113] playground - fix typos on the docs Co-authored-by: Artur Khanin <artur.khanin@akvelon.com> Merge pull request #16363 from [BEAM-13557] [Playground] show code execution time * [BEAM-13557] playground show code execution time * [BEAM-13557] fix playground reset * [BEAM-13557] playground - fix changing example * [BEAM-13557] improve null checks Merge pull request #16374 from [BEAM-13398][Playground] Split LifeCycle to DTO and business logic * [BEAM-13398][Playground] Add LifeCycleDto to separate DTO and business logic. * [BEAM-13398][Playground] Change LifeCycle structure * [BEAM-13398][Playground] Merge with master * [BEAM-13398][Playground] Merge with master Fix using `executable_files` value Fix comments * [BEAM-13398][Playground] renaming [BEAM-13616][BEAM-13646] Update vendored calcite 1.28.0 with protobuf 3.19.2 (#16473) Adds several example multi-language Python pipelines Merge pull request #16325 from [BEAM-13471] [Playground] Tag existing unit-tests * Tag java unit-test * Tag go unit-test * Tag python unit-test * Update tags * Fix checks * Edit tags * Update tags Merge pull request #16488: [BEAM-13637] Adds several example multi-language Python pipelines [BEAM-13399] Move service liveness polling to Runner type (#16487) Split builder into several builder for each step of pipeline execution [BEAM-8806] Integration test for SqsIO using Localstack Provide API to check whether a hint is known. [BEAM-13480] Increase pipeline timeout for PubSubIntegrationTest.test_streaming_data_only (#16496) Stronger typing inference for CoGBK. (#16465) Also fix empty union consistency check. [BEAM-12464] Change ProtoSchemaTranslator beam schema creation to match the order for protobufs containing Oneof fields (#14974) * ProtoSchemaTranslator now orders oneof fields in the resultant beam schema in accordance with their location in the protobuf definition * add reverse order protobuf * add noncontiguous oneof and some renaming * Comments and variable renaming * add reversed row tests * add noncontiguous tests * remove redundant null check * minor test comment update * update * add reversedonof test * add noncontiguous oneof test Co-authored-by: Reuben van Ammers <reuben.vanammers@eliiza.com.au> Introduce the notion of a JoinIndex for fewer shuffles. (#16101) This allow joining (aka zipping) operations to execute without requiring a global repartitioning as long as the operands have a common, unchanged ancestor index. Also add counter and tests to ensure expected fusion. Merge pull request #16467 from [BEAM-12164]: SpannerIO DetectNewPartitions SDF * [BEAM-12164]: SpannerIO DetectNewPartitions SDF Adds the DetectNewPartitions SDF. This component will be responsible for: - Emitting a watermark based on the min of all unfinished partitions in the metadata table. - Querying all partitions in the CREATED state. - Updating the created partitions to SCHEDULED state. - Emitting the scheduled partitions to the PCollection. This SDF will run periodically as based on the configured resume interval (default is 100ms, best effort). * chore: fix linting violations Co-authored-by: Hengfeng Li <hengfeng@google.com> [BEAM-12558] Fix doc typo. Merge pull request #16385 from [BEAM-13535] [Playground] add cancel execution button * [BEAM-13535] playground - add cancel execution button * [BEAM-13535] fix merging cancel button * [BEAM-13535] fix hash code to recommended * [BEAM-13535] cancel pipeline execution log * [BEAM-13535] playground - fix merge conflicts Merge pull request #16485 from [BEAM-13486] [Playground] For unit tests (java) if one of tests fails the output goes to stdOutput * [BEAM-13486] Added errorOutput to output tab * [BEAM-13486] Fixed tests for code repository * [BEAM-13486] Fixed runOutputError printing Merge pull request #16413 from blais/master [BEAM-13455] Remove duplicated artifacts when using multiple environments with Dataflow Java [BEAM-12164]: Add SDF for reading change stream records Adds ReadChangeStreamPartitionDoFn, which is an SDF to read partitions from change streams and process them accordingly. This component receives a change stream name, a partition, a start time and an end time to query. It then initiates a change stream query with the received parameters. Within a change stream, 3 types of records can be received: 1. A Data record 2. A Heartbeat record 3. A Child partitions record Upon receiving #1, the function updates the watermark with the record's commit timestamp and emits the record into the output PCollection. Upon receiving #2, the function updates the watermark with the record's timestamp, but it does not emit any record into the PCollection. Upon receiving #3, the function updates the watermark with the record's timestamp and writes the new child partitions into the metadata table. These partitions will be later scheduled by the DetectNewPartitions component. Once the change stream query for the element partition finishes, it marks the partition as finished in the metadata table and terminates. [BEAM-13577] Beam Select's uniquifyNames function loses nullability of Complex types while inferring schema [BEAM-13400] JDBC IO does not support UUID and JSONB PostgreSQL types and OTHER JDBC types in general * Support BLOB, CLOB, OTHER, JAVA_OBJECT JDBC types * Add native support of UUID type Merge pull request #16225 from ihji/BEAM-13455 [BEAM-13455] Remove duplicated artifacts when using multiple environm… [BEAM-12572] Run java examples on multiple runners (#16450) [BEAM-10206] Resolve go vet errors in protox package Merge pull request #16284: [BEAM-13400] JdbcIO should support UUID and JSONB PostgreSQL types and OTHER JDBC types in general [BEAM-13656] Provide an API to check whether a hint is known. #16508 Merge pull request #16477 from [BEAM-13560][Playground] Split builder into several builders for each step of the pipeline execution [BEAM-13560][Playground] Split builder into several builders for each step of the pipeline execution Merge pull request #16482 from [BEAM-13429][Playground] Add builder for preparers * Split builder into several builder for each step of pipeline execution * Add preparers builder * Rename preparators -> preparers * Small renaming at builders * Forgotten renaming [BEAM-13590] Fix abc imports from collections (#15850) Fix staticcheck errors in transforms directory Remove unnecessary fmt.Sprintf() in partition.go Replace bytes.Compare() with bytes.Equal() in test cases Replace string(buf.Bytes()) with buf.String() in coder_test.go Remove unnecessary blank identifier assignment in harness.go fix capitalized error strings in expansionx Clean up string cast of bytes in vet.go and corresponding tests Remove unnecessary fmt call in universal.go Remove tab from source. Redirecting cross-language transforms content (#16504) * redirected xlang content * updated redirect and added blurb for Java multi-lang quickstart * removing unnecessary alias from programming guide * improved opening sentence of Java multi-language pipelines page Merge pull request #16528 Remove tab from source. Merge pull request #16520: [BEAM-10206] Resolve go vet errors in protox package [BEAM-10206] Resolve go vet errors in protox package doc tweaks (#16498) [BEAM-12621] - Update Jenkins VMs to modern Ubuntu version (#16457) Co-authored-by: Giomar Osorio <95301399+GiomarWize@users.noreply.github.com> Merge pull request #16524: [BEAM-10206] Clean up some simple staticcheck warnings in Go SDK [BEAM-10206] Clean up some simple staticcheck warnings in Go SDK Bump beam container version. [BEAM-13664] Fix Primitives hashing benchmark (#16523) * [BEAM-13628] Fix Primitives hashing benchmark * staticcheck cleanup Merge pull request #16507: [BEAM-13137] Fixes ES utest size flakiness with _flush api and index.store.stats_refresh_interval=0 * [BEAM-13137] Fixes ES utest size flakiness with _flush api and index.store.stats_refresh_interval=0 Merge pull request #16468: [BEAM-8806] Integration test for SqsIO using Localstack (AWS Sdk v2) Remove jcenter repositories from gradle configuration. (#16532) JCenter has in read-only mode and appears to have degraded availability. https://blog.gradle.org/jcenter-shutdown Update GH Actions to use proper variables names and proper triggers Merge pull request #16492 from [BEAM-13641][Playground] Add SCIO SDK support on the CI/CD step [BEAM-13641][Playground] Add SCIO SDK support on the CI/CD step [BEAM-13430] Remove jcenter which will no longer contain any updates. (#16536) See https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/ There are already issues with spurious timeouts/failures. This removes the remaining references left after https://github.com/apache/beam/pull/16532 [BEAM-13616] Update com.google.cloud:libraries-bom to 24.2.0 (#16509) [BEAM-13680] Fixed code_repository (added pipelineUuid to RunCodeResult when status is "Finished") Merge pull request #16480: [BEAM-13631] Add deterministic SQS message coder to fix reading from SQS in batch mode (AWS Sdk v1) Merge pull request #16481: [BEAM-8806] Integration test for SqsIO using Localstack (AWS Sdk v1) Also bump FnAPI container. [BEAM-13616][BEAM-13645] Switch to vendored grpc 1.43.2 (#16543) Merge pull request #16529 Bump dataflow worker container versions. [BEAM-13616][BEAM-13646] Upgrade vendored calcite to 1.28.0:0.2 (#16544) Merge pull request #16545 from [BEAM-13680] [Playground] [Bugfix] Fix tests on frontend [BEAM-13680] [Playground] [Bugfix] Fix tests on frontend Merge pull request #16486 from [BEAM-13544][Playground] Add logs to examples CI/CD to see the progress * [BEAM-13544][Playground] Add logs for the CI/CD steps Fix minor issues * [BEAM-13544][Playground] update logs' messages * [BEAM-13544][Playground] Update using of `tqdm` Add `tqdm` to requirements.txt * [BEAM-13544][Playground] Add logic to replace `\t` with spaces for tag * [BEAM-13544][Playground] Change using `logger` to `logging` * [BEAM-13544][Playground] Fix according to `linter` and `yapf` [BEAM-13683] Correct SQL transform schema, fix expansion address override bug (#16551) * Update expansion service address after overrides * Make options field unexported to match SQL transform schema Update walkthrough.md (#16512) Fixed naming error that leads to a ParseException. "appId" is the name of the field, but then in the query it erroneously said id in the WHERE. [BEAM-11808][BEAM-9879] Support aggregate functions with two arguments (#16200) * [BEAM-11808] Enable two params in aggregate functions, add string_agg with delimiter * [BEAM-11808] Fix checkstyle warning * [BEAM-11808] Fix spotlessApply * [BEAM-11808] Fix checkstyle warning * [BEAM-11808] Fix spotlessApply * [BEAM-11808] Change initialization StringAgg and minor fixes * [BEAM-11808] Fix checkstyle warnings * [BEAM-11808] Add test cases for array_agg and timestamp null max,min * [BEAM-11808] Remove leftover file * [BEAM-11808] Enable resolved_literal as firts arg * [BEAM-11808] Remove tests, validate RESOLVED_LITERAL as second argument * [BEAM-11808] Add unsupportedException for delimiter as ResolvedParam * use zetasql exception * update test Co-authored-by: Kyle Weaver <kcweaver@google.com> Merge pull request #16506 from [BEAM-13652][Playground] Send examples' links to the frontend * [BEAM-13652][Playground] Add link to the PrecompiledObject * [BEAM-13652][Playground] Add a comment about `link` field Merge pull request #16322 from [BEAM-13407] [Playground] Preload fonts for the web application * [BEAM-13407] Added google fonts & licences to project * [BEAM-13407] Fixed license name in main.dart * [BEAM-13407] Removed unused font files * [BEAM-13407] Added licenses to google-fonts * [BEAM-13407] Changes in font licenses * [BEAM-13407] Fixed _getPipelineResult method of code_repository [BEAM-13665] Make SpannerIO projectID optional again (#16547) Fixes regression introduced by PR #15493 which inadvertently caused an NPE when the projectID was not specified for a SpannerIO read or write. Adds unit test for reading/writing both with and without projectID [BEAM-13015] Add state caching capability to be used as hint for runners to not duplicate cached data if the SDK can do it for user state and side inputs. (#16525) Merge pull request #16309: [BEAM-13503] Set a default value to throwWriteErrors in BulkIO constructor [BEAM-13503] Set a default value to throwWriteErrors in BulkIO constructor [BEAM-13015] Provide caching statistics in the status client. (#16495) * [BEAM-13015] Provide caching statistics in the status client. * Address PR comments. * Address PR comments. * Address PR comments by weighing the key. [BEAM-13611] Skip test_xlang_jdbc_write (#16554) Merge pull request #16370 from [BEAM-13556] playground - color and scroll tabs with new content * [BEAM-13556] playground - color and scroll tabs with new content * [BEAM-13556] playground - add enter after the processing has started log * [BEAM-13556] fix tests Merge pull request #16531 from [BEAM-13567] [playground] Handle run code validation and preparation errors Merge pull request #16533 from [BEAM-13548] [Playground] Add example description popover Merge pull request #16519 from [BEAM-13639] [Playground] Add notification to Log/Output tabs about cached example * [BEAM-13639] Added log about cached results for precompiled examples * [BEAM-13639] Fixed _getPipelineResult method of code_repository Merge pull request #16518 from [BEAM-13619] [Playground] Add loading animation to the catalog * [BEAM-13619] Added loading animation to the catalog * [BEAM-13619] Refactored loading indicator component * [BEAM-13619] Added license to loading_indicator component * [BEAM-13619] Fixed _getPipelineResult method of code_repository edited README and comments in Python multi-lang pipes examples Merge pull request #16243 from darshan-sj/feature/support-priority-spannerio - Making rpcPriority a ValueProvider in SpannerConfig * Making rpcPriority a ValueProvider in SpannerConfig * correcting build failure * correcting tests * formatting changes Allow Java 17 to be used in SDK add testing support Add more testing support for java 17 [BEAM-13683] Make cross-language SQL example pipeline (#16567) [BEAM-13688] fixed type in BPG 4.5.3 window section (#16560) Remove obsolete commands from Inventory job. (#16564) workaround for jamm Revert "Merge pull request #15863 from [BEAM-13184] Autosharding for JdbcIO.write* transforms" This reverts commit 818428ff092d41e408af38832c86538d15c95033. BEAM-13611 reactivating jdbcio xlang test Disable logging for memoization test. (#16556) Test loggers may cache logged items, extending their lifetime thus violating this test's expectations about when things will be gc'd. [BEAM-13689] Output token elements when BQ batch writes complete. Merge pull request #16472: [BEAM-13697] Add SchemaFieldNumber annotation Merge pull request #16373 from [BEAM-13515] [Playground] Hiding lines in an example that are not necessary * [BEAM-13515] Implemented structure of initial scroll in CodeField * [BEAM-13515] Changed scroll mechanics * [BEAM-13515] Moved values to consts, changed quotes to single * [BEAM-13515] Changed RegExp for Java, moved RegExp strings to consts * [BEAM-13515] Clarified variable declaration [BEAM-13093] Enable JavaUsingPython CrossLanguageValidateRunner test for dataflow runner v2 Merge pull request #16569 from Revert "Merge pull request #15863 from [BEAM-13184] Autosharding for … Revert "Merge pull request #15863 from [BEAM-13184] Autosharding for … Merge pull request #16371 from [BEAM-13518][Playground] Beam Playground quickstart page on the Beam website * [WIP] Beam Playground quickstart page * Add Beam Playground quickstart page in Get Started section * Removed trailing whitespace * Updated the content of Apache Beam Playground quickstart page * Updated Beam Playground quickstart page Update Java FnAPI beam master (#16572) [BEAM-13699] Replace fnv with maphash. (#16573) [BEAM-13693] Bump beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming timeout to 12 hours (#16576) Merge pull request #16380 from talatuyarer/BEAM-13577-select-uniquifyNames-nullablity [BEAM-13577] Beam Select's uniquifyNames function loses nullability of Complex types while inferring schema Merge pull request #16561 from [BEAM-13689] Output TableDestination elements when BQ batch writes complete. [BEAM-13689] Output TableDestination elements when BQ batch writes complete. [BEAM-10206] Remove Fatalf calls in non-test goroutines for tests/benchmarks (#16575) [BEAM-13687] Improved Spanner IO request count metrics Moved the creation of metrics counter to startbundle. [BEAM-13430] Re-add provided configuration (#16552) Merge pull request #16566: edited README and comments in Python multi-lang pipes examples Merge pull request #16514 from [BEAM-12164]: Add SDF for reading change stream records [BEAM-12164]: Add SDF for reading change stream records Merge pull request #16540 from [BEAM-13678][Playground]Update Github Action To Deploy Examples * Fix deploy of examples * Update license Merge pull request #16539 from [BEAM-13677][Playground]Update GitHub Actions to Build Playground Containers In Proper Way [BEAM-13677][Playground]Update GitHub Actions to Build Playground Containers In Proper Way Merge pull request #16546 from [BEAM-13661] [BEAM-13704] [Playground] Update tags for examples/katas/unit-tests * Add default_example field to the tags; Remove empty optional fields from tags; * Add quickstart category * Add new line in the categories.yaml Merge pull request #16369 from [BEAM-13558] [Playground] Hide the Graph tab and SCIO from SDK options * [BEAM-13558] Hid the Graph tab from output area & SCIO from SDK options * [BEAM-13515] Added clarifying comments [BEAM-10206] Add key for fields in wrapper (#16583) Merge pull request #16530 from Adding JSON support in SpannerIO and Spanner schema parser * Adding JSON support in SpannerIO and Spanner schema parser * formatting changes [BEAM-13687] Improved Spanner IO request count metrics [BEAM-13685] Enable users to specify cache directory under Interactive Beam options Add Jenkins test for Java 17 Fix jvm hex and skip errorprone Fix display data for anonymous classes fix jpms tests [BEAM-13716] Clear before creating a new virtual environment in setupVirtualenv [BEAM-13653] Make SnsIO.write topicArn optional. If provided, validate it and force usage on the Sns publish request. Merge pull request #16503: [BEAM-13653] Fix usage of SnsIO.write topicArn (AWS Sdk v2) Privacy policy update regarding Apache Beam Playground [BEAM-10897] Update the fastavro lower bound due to an issue on Windows (#16553) [BEAM-13605] Update pandas_doctests_test denylists in preparation for pandas 1.4.0 (#16571) Merge pull request #16538 from [BEAM-13676][Playground][Bugfix]Build Of Playground Applications Is Broken * Update build gradle files to fix build of applications without double dockerfile issues * Test fix Co-authored-by: Ilya <ilya.kozyrev@akvelon.com> Co-authored-by: Artur Khanin <artur.khanin@akvelon.com> Merge pull request #16582 from [BEAM-13711] [Playground] [Bugfix] Add Cloud Logging API in allow list * Add cloud logging api in allow list * Add api to allowed_list Merge pull request #16515 from [BEAM-13636] [Playground] Checking the default field on CI/CD step * Checking the default field on CI/CD step * Support an optional fields * Update proto; * Refactoring code * Change default value for default_example field * Refactoring code [BEAM-13275] Removed the explicit selenium dependency from setup The dependency can be transitively installed through needle, no need to fix it in setup. [BEAM-13275] Removed the explicit selenium dependency from setup [BEAM-13321] Fix exception with BigQuery StreamWriter TraceID. When running a BigQuery IO transform with a query and query location set, was given the exception: "java.lang.IllegalArgumentException: TraceId must follow the format of A:B. Actual:Dataflow". Documentation shows an example of using it with the format "Dataflow:job_id" which is what I changed this to: https://cloud.google.com/java/docs/reference/google-cloud-bigquerystorage/latest/com.google.cloud.bigquery.storage.v1.StreamWriter.Builder#com_google_cloud_bigquery_storage_v1_StreamWriter_Builder_setTraceId_java_lang_String_ [BEAM-10206] Deprecate unused shallow cloning functions (#16600) Bump Dataflow container versions (#16602) Merge pull request #16591 from ihji/BEAM-13716 [BEAM-13716] Clear before creating a new virtual environment in setupVirtualenv Improved multi-language pipelines section of the programming guide (#16587) * improved multi-language pipelines section of the programming guide * made changes to multi-lang pipeline content, in response to feedback * updating Beam version number for supporting no-code Java xlang transforms * updating one more Beam version number skip zetasql Get rid of unnessecary logs for BigQuery streaming writes in auto-sharding mode. [BEAM-13510] Don't retry on invalid SQS receipt handles. Support SCIO SDK via sbt projects Merge pull request #16478 from mosche/BEAM-13510-InvalidRetries Merge pull request #16478: [BEAM-13510] Don't retry on invalid SQS receipt handles (AWS Sdk v2) [BEAM-8807] Add integration test for SnsIO.write (Sdk v1 & v2) [BEAM-13736] Make lifting cache exact. (#16603) Merge branch 'master' of github.com:MarcoRob/beam into BEAM-12650 Merge pull request #16565 from [BEAM-13692][Playground] Implement method to receive Graph * [BEAM-13692][Playground] Add `GRAPH` subKey Add `GetGraph` api method Update `GetValue` cache method to correctly receive graph from the cache * [BEAM-13692][Playground] fixes according to `staticcheck` * [BEAM-13692][Playground] fix test * [BEAM-13692][Playground] Fix error with `RunError` output * [BEAM-13692][Playground] Change graph from []byte to string * [BEAM-13692][Playground] Edit comment * [BEAM-13692][Playground] Update comments Co-authored-by: Ilya <ilya.kozyrev@akvelon.com> Merge pull request #16502 from [BEAM-13650][Playground] Add link for examples * [BEAM-13650][Playground] Add link for examples. * [BEAM-13650][Playground] Change using string concatenation Fix according to `linter` and `yapf` * [BEAM-13650][Playground] Change the `link` field add pending jobs to readme Update README.md removed comment [BEAM-13310] remove call to get offset consumer config, which was rep… (#16588) * [BEAM-13310] remove call to get offset consumer config, which was replacing the intended group id with a randomly generated group id for commits * [BEAM-13310] run spotless commit * [BEAM-13310] fix checkstyle error Merge pull request #16599: [BEAM-13321] Fix exception with BigQuery StreamWriter TraceID. Merge pull request #16594 from [BEAM-13710][Playground] Update the Apache Beam privacy policy with Playground note [BEAM-13710][Playground] Update the Apache Beam privacy policy with Playground note [BEAM-11648] Share thread pool across RetryManager instances. RetryManager's have short lifetime (for example processElement). This means that many different threads are created instead of effectively reusing them. This is adds expense but also can result in many threads because the each individual thread lasts for up to 60 seconds and until the GC runs. Merge pull request #16595З: [BEAM-8807] Add integration test for SnsIO.write (Sdk v1 & v2) [BEAM-13737][Playground] Update logic of receiving precompiled objects; Increase timeout; added GitHub example references to Python multilang quickstart [BEAM-13746] Fix deserialization of SSECustomerKey for AWS Sdk v2 [BEAM-7928] Allow users to specify worker disk type for Dataflow runner (#16622) Merge pull request #16623: [BEAM-13746] Fix deserialization of SSECustomerKey for AWS Sdk v2 Exclude per-key order tests on Twister2 runner Merge pull request #16534 from [BEAM-13671][Playground] Add backend contribute guide to the project * [BEAM-13671][Playground] Add CONTRIBUTE.md file * [BEAM-13671][Playground] Update CONTRIBUTE.md [BEAM-13271] Bump errorprone to 2.10.0 (#16231) * Bump errorprone to 2.10.0 * Bump autoservice to 1.0.1 Works around https://github.com/google/error-prone/issues/2745 for some Beam projects * Address new check failures (non-controversial) * Address new check failures (potentially controversial) * Disable some new checks * Disable a few more checks * Address check failures in test code * Add license reference for jgit * Explicitly ignore some return values in CoGbkResultTest * Update sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Regex.java Co-authored-by: kileys <kileysok@gmail.com> * Update SnsIOTest Co-authored-by: kileys <kileysok@gmail.com> [BEAM-13595] Don't load main session when cloudpickle is used. (#16589) Merge pull request #15767 from ihji/BEAM-13093 [BEAM-13093] Enable JavaUsingPython CrossLanguageValidateRunner test … Update readme for XVR tests Merge pull request #16626 from ihji/update_readme Update readme for XVR tests [adhoc] Test S3Options and AwsOptions for Sdk v2 [BEAM-13537] Fix NPE in kafkatopubsub example (#16625) Merge pull request #16628: [adhoc] Test S3Options and AwsOptions for Sdk v2 [BEAM-13740] update java_tests.yml to remove setup-go, which is misconfigured and blocking the build (#16608) Fix google3 import error Merge pull request #16604: [BEAM-13714] Get rid of unnecessary logs for BigQuery streaming writes [BEAM-12976] Implement Java projection pushdown optimizer. (#16513) * finish basic pushdown optimizer implementation * add docstrings * add tests for non pbegin inputs * support pushdown on producers with multiple outputs; ignore producers with inputs. * make test code private * fix null checking * use beam null check instead of java util * vendored guava * nit: dont need to use string format * address review comments Make num-stages counter into an internal counter. This unbreaks tests that (arguably to brittly) reject extra counters. Merge pull request #16635 from ihji/fix_google3_import Fix google3 import error due to missing Python gradle build file [BEAM-13751] Don't block on gcloud when attempting to get default GCP region. Merge branch 'master' of github.com:apache/beam into java7tests [BEAM-13751] Parameterize wait timeout so test doesn't waste 2s. [BEAM-13751] Add comment explaining sleep. Merge pull request #16621: added GitHub example references to Python multilang quickstart Merge pull request #16579 from Revert "Revert "Merge pull request #15863 from [BEAM-13184] Autoshard… * Revert "Revert "Merge pull request #15863 from [BEAM-13184] Autosharding for JdbcIO.write* transforms"" This reverts commit 421bc8068fc561a358cfbf6c9842408672872120. * Using batchSize to define element batch size * Handle corner case for null list Merge pull request #16606 from [BEAM-13247] [Playground] Embedding iframe * [BEAM-13247] Building embedded iframe with run functionality * [BEAM-13247] Implemented Run button & log section to the embedded iframe * [BEAM-13247] Refactored components * [BEAM-13247] Restricted editing in embedded iframe * [BEAM-13247] Implemented editable/not editable versions of iframe, added Output/Log TabBar * [BEAM-13247] Redesigned page providers for EmbeddedPlaygroundPage, changed default ratio of the EmbeddedSplitView * [BEAM-13247] Fixes after merge * [BEAM-13247] Added possibilities to change height & use any text in iFrame, removed codeScrolling. Added new iFrame on the try-beam-playground page * [BEAM-13247] Changed iFrame src link * [BEAM-13247] Fixed PR remarks Update Python SDK beam-master tags (#16630) Merge pull request #16592 from [BEAM-13722][Playground] Add precompiling of the graph into examples CI/CD * [BEAM-13722][Playground] Add receiving graph value from backend to send it as a metadata field * [BEAM-13722][Playground] Add license * [BEAM-13722][Playground] Regenerate proto files * [BEAM-13722][Playground] Remove graph from meta file; Add sending of graph as a separate file; * [BEAM-13722][Playground] add condition for SDK * [BEAM-13722][Playground] Regenerate proto files * [BEAM-13722][Playground] Regenerate proto files * [BEAM-13722][Playground] small fix * [BEAM-13722][Playground] add try-except to receive graph Merge pull request #16505 from [BEAM-13527] [Playground] Pipeline options dialog * [BEAM-13527] pipeline options dropdown * [BEAM-13527] playground - parse pipeline error message * [BEAM-13527] playground - fix parse options * [BEAM-13527] playground - move pipelines options lines count to const * [BEAM-13527] playground fix tests * fix merge * [BEAM-13527] pipeline options fix review comments * [BEAM-13527] pipeline options fix review comments spotless spotless Generalize S3FileSystem to support multiple URI schemes. This ports https://github.com/apache/beam/pull/15036 to aws2. [BEAM-13768] Fix NullPointerException in BigQueryStorageSourceBase Fix Java SDK container image name for load-tests and nexmark Merge pull request #16639 from ibzib/BEAM-13751 [BEAM-13751] Don't block on gcloud when attempting to get default GCP… [BEAM-13293] XLang Jdbc IO for Go SDK (#16111) [BEAM-10206] Add Go Vet to Github Actions (#16612) Merge pull request #16607: [BEAM-13245] Generalize S3FileSystem to support multiple URI schemes. Change executable name fo go tests Fix java test [BEAM-13769] Skip test_main_session_not_staged_when_using_cloudpickle (#16651) Merge pull request #16663 from [BEAM-13773] [Playground] [Bugfix] Error during executing of tests on Java SDK [BEAM-13773] [Playground] [Bugfix] Error during executing of tests on Java SDK Merge pull request #16662 from [BEAM-13772][Playground][Bugfix] Change executable name for go tests [BEAM-13772][Playground][Bugfix] Change executable name for go tests [BEAM-6744] Support implicitly setting project id in Go Dataflow runner (#16643) Merge pull request #16493 from [BEAM-13632][Playground] Save catalog data to the cache * Added saving of precompiled objects catalog to cache at the server startup * Added caching of the catalog only in case of unspecified SDK * Update regarding comments * Update regarding comments * Simplified logging regarding comment * Updates regarding comments * Update for environment_service_test.go * Docstring update for GetPrecompiledObjects Exclude jul-to-slf4j from Spark runner in quickstart POM templates (#16656) [BEAM-13734] Support cache directories that use GCS buckets [BEAM-11936] Enable a few errorprone checks that were broken by pinned guava in annotationProcessor (#16647) * Don't pin old guava for annotationProcessor configurations * Enable broken checks * Fix CassandraIOTest assertion [BEAM-13780] Add CONTRIBUTING.md pointing to main guide (#16666) Co-authored-by: Danny McCormick <damccorm@github.com> Merge pull request #16624 from [BEAM-13749] Exclude per-key order tests on Twister2 runner [BEAM-13749] Exclude per-key order tests on Twister2 runner [BEAM-13777] Accept cache capacity as input parameter instead of default max integer (#16652) * Change cache capacity to suggested default value instead of max integer to prevent out of memory error * Receive cache capacity as input prarmeter * Spotless fixes * [BEAM-13777] keep .of methods compatibility and use suggested default value for cache capacity by confluent [BEAM-13051][A] Enable pylint warnings (function-redefined/bad-super-call) (#16521) Merge pull request #16563 from [BEAM-13701][Playground] Support SCIO SDK via sbt projects [BEAM-13701][Playground] Support SCIO SDK via sbt projects [BEAM-13779] Fix pr labeling (#16665) Co-authored-by: Danny McCormick <damccorm@github.com> Merge pull request #16581 from [BEAM-12164]: Add SpannerIO.readChangeStreams * feat: add experimental spanner readChangeStreams Adds the SpannerIO.readChangeStreams feature that will enable users to consume a change stream from Cloud Spanner. This feature is under preview now, and can only be used for allowlisted customers. When reading a change stream the users will be able to operate on a PCollection of DataChangeRecords, containing the modifications made to the database as well as the type of operation. * fix: remove public api exposure of opencensus Do not expose the Opencensus TraceSampler in the SpannerIO.readChangeStreams. This is done so that we can upgrade the opencensus library without having to concern ourselves with which version customers are using. This commit also removes the deserializer option since it is not used. Co-authored-by: Zoe Cai <zoc@google.com> Fix labeler trigger (#16674) Merge pull request #16619: [BEAM-11648] Share thread pool across RetryManager instances. Merge pull request #16645 from ihji/fix_nexmark_dataflow_v2 Fix Java SDK container image name for load-tests and nexmark [BEAM-13781] Exclude grpc-netty-shaded from gax-grpc's dependency (#16672) [BEAM-13051] Fixed pylint warnings : raising-non-exception (E0710), super-on-old-class (E1002) (#16541) * [BEAM-BEAM-13051] Fixed pylint warnings unexpeted-keyword-arg (E1123), undefined-variable (E0602) * Delete raising-non-exception and super-on-old-class from .pylintrc * Disable checks that broke tests * Delete extra blanck lines [BEAM-13740] Correctly install go before running tests (#16673) [BEAM-12830] Update local Docker env Go version. (#16670) [BEAM-13051][B] Enable pylint warnings (import-error/invalid-unary-operand-type) (#16522) * [BEAM-13051] Pylint invalid-unary-operand-type warning enabled * [BEAM-13051] Added unary negative operator to Timestamp * [BEAM-13051] Disabled invalid-unary-operand-type warning in specific cases [BEAM-13430] Revert Spark libraries in Spark runner to provided (#16675) [BEAM-12240] Add Java 17 support (#16568) Merge branch 'master' of github.com:apache/beam into java7tests [BEAM-13760] Add random component to default python dataflow job name (#16641) * Add random component to default python dataflow job name * Correct formattiing of multiline statements * Remove uuid from random string implementation. Fix trigger Merge pull request #16655 from [BEAM-12164]: Add retry protection to DetectNewPartitions * feat: add experimental spanner readChangeStreams Adds the SpannerIO.readChangeStreams feature that will enable users to consume a change stream from Cloud Spanner. This feature is under preview now, and can only be used for allowlisted customers. When reading a change stream the users will be able to operate on a PCollection of DataChangeRecords, containing the modifications made to the database as well as the type of operation. * fix: remove public api exposure of opencensus Do not expose the Opencensus TraceSampler in the SpannerIO.readChangeStreams. This is done so that we can upgrade the opencensus library without having to concern ourselves with which version customers are using. This commit also removes the deserializer option since it is not used. * [BEAM-12164]: Add retry protection to DetectNewPartitions The original algorithm of the DetectNewPartitions is susceptible to failures, because it produces side effects on every try which is not idempotent. Specifically, it marks the partitions as SCHEDULED in the Spanner database and outputs them. If there is a bundle commit failure, during retry, the already SCHEDULED partitions will not be picked up again. We change the algorithm in this PR to always schedule partitions that have a created at timestamp greater than the one saved in the DetectNewPartitions restriction. When scheduling the partitions, this SDF will also claim the created at of such partitions, advancing the timestamp saved. If there is a bundle commit failure, the restriction timestamp won't be saved, thus the partitions in the bundle will be picked up again regardless of their state. Co-authored-by: Zoe Cai <zoc@google.com> Merge pull request #16586 from [BEAM-13731] FhirIO: Add support for BATCH bundle errors. * FhirIO: Add support for BATCH bundle errors. * SpotlessApply skip checker framework fix app name [BEAM-13011] Adds a link to Multi-language Pipelines Tips wiki page (#16649) * Adds a link to Multi-language Pipelines Tips wiki page * Address reviewer comment * Add anchor [BEAM-13734] Support cache directories that use GCS buckets remove duplicate property check [BEAM-12572] Run python examples on multiple runners (#16154) [BEAM-13574] Large Wordcount (#16455) Avoid packaging avro in the java harness jar. Unfortunately we can't simply shade it, as it's used in the public API (AvroCoder) of sdks:java:core. However, outside that use, it should not be needed when running in portable mode. Users will have to provide their own avro (typically by depending on sdks:java:core, but possibly choosing another version explicitly or from elsewhere). [BEAM-13293] Refactor JDBC IO Go Wrapper (#16686) * [BEAM-13293] refactor jdbcio * [BEAM-13293] resolve comments Edit license script for Java, add manual licenses for xz (#16692) [BEAM-13563] Restructure Kinesis Source for Aws 2 internally to prepare for ClientBuilderFactory Merge pull request #16657: [BEAM-13563] Restructure Kinesis Source for AWS 2 [BEAM-4665] Allow joining a running dataflow pipeline without throwing (#16689) [BEAM-13801] Add standard coder tests for state backed iterable. (#16696) Add coverage for Java and Python. Skip testing for Go. Go change a future follow-up. Merge branch 'master' of github.com:apache/beam into java7tests Merge pull request #16644: [BEAM-13768] Fix NullPointerException in BigQueryStorageSourceBase [BEAM-13430] Fix provided configuration by removing extendsFrom for implementation (#16688) [BEAM-12830] Print clearer go version fail message (#16693) Add reference to Books to Learning Resources in website Use ThreadLocal for DESERIALIZATION_CONTEXT (#16680) Co-authored-by: Kellen Dye <kellend@spotify.com> Merge pull request #16682: Add reference to Books to Learning Resources in website Minor: Add apt update after adding deadsnakes repository in dev env (#16708) Merge pull request #16638 from robertwb/internal-counters Make num-stages counter into an internal counter. Merge pull request #16694 [BEAM-13800] Avoid packaging avro in the java harness jar. [BEAM-13807] Regenerate container images to get TF 2.8.0 (#16707) * Regenerate container images to get TF 2.8.0 * Add tf-estimator-nightly license URL [BEAM-13399, BEAM-13683] Eagerly materialize artifacts for automated expansion service, add feature to SQL transform (#16671) Merge pull request #16617 from [BEAM-13743] [Playground] Add context line for examples * Add context line for examples * Subtract number of tag lines from context_line * Update proto * Update licenses; Merge pull request #16618 from [BEAM-13744] [Playground] Add context line field to the tags * Add context_line to tags * Edit context_line`s in the tags * Edit info about default_example Merge pull request #16698 from [BEAM-13802][Playground] [Bugfix] Clean all build directroies, clean kubernetes env [BEAM-13293][BEAM-13806] Pipe a SchemaIO flag through Go integration test script. (#16705) This is needed for integration tests of Schema IO based xlang transforms, like JDBC or BigQuery. Merge pull request #16620 from [BEAM-13737][Playground] Local retrieving of example objects may fail with context deadline [BEAM-13737][Playground] Local retrieving of example objects may fail with context deadline [BEAM-13605] Modify groupby.apply implementation in preparation for pandas 1.4.0 (#16706) * Modify groupby.apply implementation in preparation for pandas 1.4.0 * fixup! Modify groupby.apply implementation in preparation for pandas 1.4.0 * Address review comments Merge pull request #16436 from [BEAM-1330] - DatastoreIO Writes should flush early when duplicate keys arrive * Switching to a HashSet to flush batch when duplicate is found * Keeping track of duplicates in a separate HashSet * clearing HashSet when flushing the data * Adding test for the duplicated entries scenario * Changing test implementation to use TestPipeline * Applying spotless * Executing tests with Mutations * Forcing the processing of the List of mutation in the same order the mutations were added to the original List * Moving the batchSize.update call to within the flushBatch method * Bumping to 7 since batchSize.update is now being called from within flushBatch [BEAM-13813] Add support for URL artifact to extractStagingToPath [BEAM-13663] Remove unused duplicate option for AWS client configuration (AWS Sdk v1). Remove Python SQL Test example from catalog [BEAM-13246] Add support for S3 Bucket Key at the object level (AWS Sdk v2). Fix timer consistency in direct runner [BEAM-13757] adds pane observation in DoFn (#16629) Merge pull request #16650:[BEAM-11971] Fix directrunner timer consistency Change links to Books from Amazon to Publisher [BEAM-13605] Add support for pandas 1.4.0 (#16590) * Addding new functions to / fixing doctests * Add _rename and value_counts() * Move import statement * Add if DataFrame has value_counts attr * Fix typo * Update precommit script and setup.py to 1.4 * Add backwards compatability for rename and replace * Add docstring and simplify kwargs * Skip DataFrame construction with series * Add change to CHANGES.md * Skip failing pyarrow test * Add pandas 1.4 to tox.ini Merge pull request #16718: [website] Change links to Books from Amazon to Publisher [BEAM-13761] adds Debezium IO wrapper for Go SDK (#16642) [BEAM-12976] Log projection pushdown optimizations. Merge pull request #16640 from kileys/java7tests [BEAM-12240] Add Jenkins tests for Java 17 [BEAM-13024] Unify PipelineOptions behavior (#16719) Update sdks/go/pkg/beam/artifact/materialize_test.go Co-authored-by: Robert Burke <lostluck@users.noreply.github.com> Merge pull request #16710 from ihji/BEAM-13813 [BEAM-13813] Add support for URL artifact to extractStagingToPath Merge pull request #16713 from [BEAM-13815] [Playground] [Bugfix] Remove Python SQL Test example from catalog [BEAM-13815] [Playground] [Bugfix] Remove Python SQL Test example from catalog Merge pull request #16605 from [BEAM-13634][Playground] Create a separate Dockerfile for the routing service * Added saving of precompiled objects catalog to cache at the server startup * Added caching of the catalog only in case of unspecified SDK * Update regarding comments * Update regarding comments * Simplified logging regarding comment * Updates regarding comments * Update for environment_service_test.go * Docstring update for GetPrecompiledObjects * Add router services as separate container * too stage container builder * add empty string to build_playground_backend.yml * fix comments Co-authored-by: Artur Khanin <artur.khanin@akvelon.com> Co-authored-by: akustov <akustov@diasoft.ru> Co-authored-by: Ilya <ilya.kozyrev@akvelon.com> Merge pull request #16593 from [BEAM-13725][Playground] Add graph to the precompiled objects * [BEAM-13725][Playground] Add graph field to `PrecompiledObject` * [BEAM-13725][Playground] Fix test * [BEAM-137235][Playground] Remove `graph` from PrecompiledObject Add a new methods to receive graph of the PrecompiledObject. * [BEAM-13725][Playground] Update comments; Remove unnecessary field; * [BEAM-13725][Playground] Fix tests * [BEAM-13725][Playground] Regenerate proto files * [BEAM-13725][Playground] renaming * [BEAM-13725][Playground] Regenerate files * [BEAM-13725][Playground] Regenerate proto files Merge pull request #16699 from [BEAM-13789][Playground] Change logic of keeping examples to the bucket on CD side * [BEAM-13789][Playground] Change examples' folders structure Fix by `yapf` and `pylint` * [BEAM-13789][Playground] Change examples' folders structure Fix by `yapf` and `pylint` Fixed CSS for Case study page Merge pull request #16730: Fixed CSS for Case study page [BEAM-13203] Deprecate SnsIO.writeAsync for AWS Sdk v2 due to risk of data loss. Merge pull request #16712: [BEAM-13663] Remove unused duplicate option for AWS client configuration (AWS Sdk v1) Merge pull request #16711: [BEAM-13203] Deprecate SnsIO.writeAsync for AWS Sdk v2 due to risk of data loss. [BEAM-13828] Fix stale bot (#16734) Merge pull request #16364 from [BEAM-13182] Add diagrams to backend readme * Add diagrams to backend README.md * Fix comments * Fix comments [BEAM-13811] Fix save_main_session arg in tests examples (#16709) …

sketching the core API for JS SDK

* [Tour of Beam][Frontend][#22600] TourScreen layout * theme setup * Replaced ThemeProvider with ThemeSwitchNotifier * header with theme mode switcher and logo * page container with header & footer * theme mode tests * renamed the directory to tour-of-beam * compressed beam_logo.png * added missing license comments * rudimentary layout of the first screen * review comments fixes #1 * moved notifyListeners inside then * responsive todo * split into 2 simple functions * deleted redundant constants & replaced 2018 text theme with 2021 * styling refinement * home screen layout * clickable sign in text * font weights fix * removed _getBaseFontTheme function * fixed border and bg color * color fixes * difficulty component * _LastModuleBody * todo in test * footer border * fixed overflows * replaced Project prefix with Tob * replaced then with await * inferred type * started translation of the home screen * sorted translations * Complexity comments * comment fixes * home screen translations * sign in overlay * import fix * integration test does not fail * playground_components package with dismissible_overlay * missing license * removed _dots from build * widgets refinement * renamed home screen to welcome screen * deleted copyWith * _SdkButton * trailing comma & pubspec formatting * license and lints * license * removed license from .metadata * pubspec formatting * total lints update * changed from tour_of_beam to tour-of-beam in build.gradle.kts * license check * _SdkButton mimics Radio button * renamed MyApp to TourOfBeamApp * onChanged mimics Radio button Tour of Beam frontend blank project [Tour of Beam][Frontend][#22600] TourScreen layout TourScreen layout (#22600) common theme, constants, split view missing license flutter_gen, summary layout details content layout details no functional widgets in split view main screen todos & translation main screen todos & translation comment fixes #1 ExpansionTileWrapper SplitViewController lists in tour screen widgets comment fixes #1 (31.08) split view package in PGC fixed button overflow splitter theme color comment fixes #2 (31.08) gradlew check welcome screen overflow test (#22600) SDK dropdown (#22600) flexible complete unit OutlinedButton (#22600) renamed PageContainer to TobScaffold dropdown style refinement DropdownButton implicit type sdk instead of e licenses #22600 renamed _ShrinkedTour to _NarrowTour #22600 tour screen style refinement #22600 BeamDivider in PGC #22600 removed todo, added license #22600 built with text #22600 _WideWelcome with IntrinsicHeight (#22600) Co-Authored-By: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com> * addressing review comments #22600 replaced magic numbers #22600 comments (#22600) comments #22600 comments #22600 comments #22600 comments #22600 comments, flutter 3.3.0 upgrade #22600 renamed ActionPadding to ActionVerticalPadding #22600 actions formatting #22600 * branded sign in buttons #22600 * _BrandedSignInButtons #22600 * _Divider color #22600 * profile #22600 * moved split_view from PGC into ToB #22600 * indentation fix #22600 * split ProfileContent into widgets #22600 * Extract playground components to a separate package (#22600) * Minor fixes (#22600) * Address review issues (#22600) * Upgrade Flutter to v3.3.2 (#22600) * Add precommit Gradle task for playground_components, add code generation to frontend Gradle task, remove generated mocks, fix linter issues (#22600) * startTour button (#22600) * lint fixes (#22600) * Fix highlighting for Python and SCIO (#22600) Co-authored-by: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com>

* learning content for introduction module * removing white spaces from md files * delete whitespaces * delete whitespaces in python * delete whitespace #2 * divide pipeline concepts * add pipeline example concepts * adding category tag to python examples * adding category to java examples * adding category to go examples * fixed go example * fixed go example compilation * fixing python duplicate example names * add runner concepts * fixing java examples * add licence for runner unit * some minor fixes for unit names * fixed unit name * resolving CR comments * adding complexity to examples * adding tags * fixed go example compilation * fixed python example with duplicate transform * change indent python * fixing missing pipeline options * change arrow symbol * delete example prefix * minor formatting and readability fixes * add example description * minor fix * minor code review comment Co-authored-by: Abzal Tuganbay <abzal.tugan@gmail.com>

# This is the 1st commit message: Add map_windows support to Go SDK Adds support to the Go SDK for the [map_windows][1] urn. The existing type model does not allow `IntervalWindow` as a FullValue and the existing coders will throw away the window values when decoding the KV type [here][2] in `elideSingleElmFV`. [1]: https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto#L296 [2]: https://github.com/camphillips22/beam/blob/8be4cdcaf65a1e53e3041ac3354e2e99c845e915/sdks/go/pkg/beam/core/runtime/exec/coder.go#L580-L580 # The commit message apache#2 will be skipped: # rename # The commit message apache#3 will be skipped: # unmarshal into function spec # The commit message apache#4 will be skipped: # type switch # The commit message apache#5 will be skipped: # use data window

…lection (#24457) * Content tree navigation (#23593) Unit content navigation (#23593) Update URL on node click (#23593) Active unit color (#23593) removeListener in unit (#23593) First unit is opened on group title click (#23593) WIP by Alexey Inkin (#23593) selectedUnitColor (#23593) Unit borderRadius (#23593) RegExp todo (#23593) added referenced collection package to remove warning (#23593) small refinement (#23593) expand on group tap, padding, openNode (#23593) group expansion bug fix (#23593) selected & unselected progress indicators (#23593) * AnimatedBuilders instead of StatefulWidgets in unit & group (#23593) * fixed _getNodeAncestors (#23593) * get sdkId (#23593) * addressing comments (#23593) * sdkId getter & StatelessExpansionTile (#23593) * expand & collapse group (#23593) * StatelessExpansionTile (#23593) * license (#23593) * ValueChanged and ValueKey in StatelessExpansionTile (#23593) * optional SDK selector in tour scaffold * AppNotifier * moved AppNotifier into _initializeState * StorageKeys * remove listener * auth, complete unit, user progress AuthNotifier draft (#23692) Comments (#23692) Comments (#23692)(1) sign in with google works (#23692) new configs (#23692) get user progress draft (#23692) comment fixes (#23692) sign in in IntroTextBody (#23692) reverted config (#23692) comment fixes (#23692) WIP before rebase (merge) (#23692) Squashed commit of the following: commit bff4919ff00ec3b5d7186efde41c884dfc4c8344 Merge: 79ba69483a ce8d618c77 Author: Alexey Romanenko <33895511+aromanenko-dev@users.noreply.github.com> Date: Thu Nov 17 10:34:02 2022 +0100 Merge pull request #24186: Uses _all to follow alias/datastreams when estimating index size commit 79ba69483a84ea0278d0b0ddb141200739607c77 Merge: 245fea9040 b7e860a762 Author: Chamikara Jayalath <chamikaramj@gmail.com> Date: Wed Nov 16 20:47:40 2022 -0800 Merge pull request #24218: Update Python wheel format for RC validation commit 245fea904014cd58d4148807463dbaa40000774c Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed Nov 16 18:12:33 2022 -0800 Bump loader-utils from 1.4.1 to 1.4.2 in /sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel (#24191) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit e1de8e78deeb5d17617fda6591429eaaf8abb8a2 Author: Yi Hu <yathu@google.com> Date: Wed Nov 16 20:48:06 2022 -0500 Fix PythonLint (#24219) commit b7e860a7621771c300dcec625655f87e62591323 Author: Chamikara Jayalath <chamikara@apache.org> Date: Wed Nov 16 17:28:31 2022 -0800 updates commit c2feb09ea49dd815b69c65e531ce34128756d988 Author: Chamikara Jayalath <chamikara@apache.org> Date: Wed Nov 16 17:06:08 2022 -0800 updates commit ce8d618c77d23e20a1ddb128bb8183048597d096 Author: egalpin <egalpin@users.noreply.github.com> Date: Wed Nov 16 16:43:57 2022 -0800 Adds test for following aliases when estimating index size commit 959719d01c627328c0ca2849d2b7e2c9b322d4d1 Author: Chamikara Jayalath <chamikara@apache.org> Date: Wed Nov 16 15:16:06 2022 -0800 Temporary update Python RC validation job commit b952b41788acc20edbe5b75b2196f30dbf8fdeb0 Author: Yi Hu <yathu@google.com> Date: Wed Nov 16 14:18:12 2022 -0500 Python TextIO Performance Test (#23951) * Python TextIO Performance Test * Add filebasedio_perf_test module for unified test framework for Python file-based IOs * Fix MetricsReader publishes metrics duplicately if more than one load test declared. This is because MetricsReader.publishers was static class variable * Fix pylint * Distribute Python performance tests random time at a day instead of all at 3PM * Add information about length conversion commit 017f2cbde124af40a43be99ec88289fcf63c1c95 Merge: fef8acdbc0 88dba4f494 Author: Chamikara Jayalath <chamikaramj@gmail.com> Date: Wed Nov 16 10:39:52 2022 -0800 Merge pull request #24187: Add a reference to Java RunInference example commit fef8acdbc0ecbcc85b49144adaf8830e3bc6b2de Merge: 6e9187e67e ead245539d Author: Ahmet Altay <aaltay@gmail.com> Date: Wed Nov 16 10:24:53 2022 -0800 Merge pull request #24199 from Laksh47/issue#24196 refs: issue-24196, fix broken hyperlink commit 6e9187e67e1bd8f73997f437f0ed4c29880ed73b Author: Darkhan Nausharipov <31556582+nausharipov@users.noreply.github.com> Date: Wed Nov 16 22:33:50 2022 +0600 [Tour of Beam] [Frontend] Content tree URLs (#23776) * Content tree navigation (#23593) Unit content navigation (#23593) Update URL on node click (#23593) Active unit color (#23593) removeListener in unit (#23593) First unit is opened on group title click (#23593) WIP by Alexey Inkin (#23593) selectedUnitColor (#23593) Unit borderRadius (#23593) RegExp todo (#23593) added referenced collection package to remove warning (#23593) small refinement (#23593) expand on group tap, padding, openNode (#23593) group expansion bug fix (#23593) selected & unselected progress indicators (#23593) * AnimatedBuilders instead of StatefulWidgets in unit & group (#23593) * fixed _getNodeAncestors (#23593) * get sdkId (#23593) * addressing comments (#23593) * sdkId getter & StatelessExpansionTile (#23593) * expand & collapse group (#23593) * StatelessExpansionTile (#23593) * license (#23593) * ValueChanged and ValueKey in StatelessExpansionTile (#23593) Co-authored-by: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com> Co-authored-by: Alexey Inkin <alexey.inkin@akvelon.com> commit b33fac2aa533d77cfa47f88466c8cd6bd3f3e864 Author: Bruno Volpato <bvolpato@google.com> Date: Wed Nov 16 10:51:11 2022 -0500 Use only ValueProviders in SpannerConfig (#24156) commit 5f013ab6567ec75b460b2081d7f89d332320caff Author: Robert Burke <lostluck@users.noreply.github.com> Date: Wed Nov 16 07:23:10 2022 -0800 revert upgrade to go 1.19 for action unit tests (#24189) commit 9337f4dbecc929886f8559949a082a649fd9d1bb Author: Yi Hu <yathu@google.com> Date: Wed Nov 16 10:18:42 2022 -0500 Fix Python PostCommit Example CustomPTransformIT on portable (#24159) * Fix Python PostCommit Examples on portable * Fix custom_ptransform pipeline options gets modified * Specify flinkConfDir commit ead245539d01dec0f3e08699c1e1cc6777a5ef0e Author: Laksh <lakshmanansathya@gmail.com> Date: Wed Nov 16 09:32:46 2022 -0500 refs: issue-24196, fix broken hyperlink commit e83a996d4374d467d95bcfad7166905622ec615c Merge: 2fc56ec663 ffdee0b6ed Author: Jan Lukavský <je.ik@seznam.cz> Date: Wed Nov 16 15:15:31 2022 +0100 Merge pull request #24192: Re-use serializable pipeline options when already available. commit ffdee0b6edb8638c78a65ec85c727ea5dde1cb2f Author: Jozef Vilcek <jvilcek@zetaglobal.com> Date: Mon Nov 14 16:48:18 2022 +0100 Re-use serializable pipeline options when already available (#24192) commit 88dba4f494829b2b3530b767fb8c5252e0d2ba44 Author: Chamikara Jayalath <chamikaramj@gmail.com> Date: Tue Nov 15 16:21:22 2022 -0800 Add a reference to Java RunInference example commit 2fc56ec663e335cfcf37dc57d471f79b601414f4 Merge: f763186987 83f1bc19b9 Author: Kenn Knowles <kenn@apache.org> Date: Tue Nov 15 16:16:47 2022 -0800 Merge pull request #24142: Fix arguments to checkState in BatchViewOverrides commit f763186987c00ba1d26efdc35406436a1fa69a9a Merge: c2bc2135e9 0d7ca04182 Author: Ning Kang <ningkang0957@gmail.com> Date: Tue Nov 15 15:25:20 2022 -0800 Addresses #24161 Updated README of Interactive Beam commit c2bc2135e9bce715990a5d5551e2bc2dc0311da4 Author: Doug Judd <nuggetwheat@gmail.com> Date: Tue Nov 15 14:48:26 2022 -0800 Strip FGAC database role from changestreams metadata requests (#24177) Co-authored-by: Doug Judd <nuggetwheat@google.com> commit af637974f96ad1b5110d7dea3f9a26c68e19a51b Author: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Date: Tue Nov 15 17:16:43 2022 -0500 Add custom inference function support to the PyTorch model handler (#24062) * Initial type def and function signature * [Draft] Add custom inference fn support to Pytorch Model Handler * Formatting * Split out default * Remove Keyed version for testing * Move device optimization * Make default available for import, add to test classes * Remove incorrect default from keyed test * Keyed impl * Fix device arg * custom inference test * formatting * Add helpers to define custom inference functions using model methods * Trailing whitespace * Unit tests * Fix incorrect getattr syntax * Type typo * Fix docstring * Fix keyed helper, add basic generate route * Modify generate() to be different than forward() * formatting * Remove extra generate() def commit a014637106970a0a0e9eb7944aa5caf79fa5fd37 Author: egalpin <egalpin@users.noreply.github.com> Date: Tue Nov 15 13:57:54 2022 -0800 Uses _all to follow alias/datastreams when estimating index size Fixes #24117 commit 0d7ca041823bc2b09f76f86fdfd1d0b9508c9c88 Author: Ning Kang <ningkang0957@gmail.com> Date: Tue Nov 15 13:57:27 2022 -0800 Minor update commit e8fc759d756f4a987e41d2b9da56b906a6cd7736 Author: Ning Kang <ningkang0957@gmail.com> Date: Tue Nov 15 13:52:18 2022 -0800 Updated README of Interactive Beam Removed deprecated cache_dir runner param in favor of the cache_root global option. commit 08d5f72e5f35d41f3e9fa9fe799caea6bed1b7a7 Author: Anand Inguva <34158215+AnandInguva@users.noreply.github.com> Date: Tue Nov 15 16:34:21 2022 -0500 [Python]Support pipe operator as Union (PEP -604) (#24106) Fixes https://github.com/apache/beam/issues/21972 commit 526e7a58b62682582c27173ab21ed8667ddab766 Author: Scott Strong <scott.strong87@gmail.com> Date: Tue Nov 15 16:26:45 2022 -0500 Using Teardown context instead of deprecated finalize (#24180) * Using Teardown context instead of deprecated finalize * making function public Co-authored-by: Scott Strong <scott.strong@wunderkind.co> commit fb4d1d4dea7b26ed538a9f6aca0ed41e8c300e37 Author: Danny McCormick <dannymccormick@google.com> Date: Tue Nov 15 16:25:22 2022 -0500 Fix broken json for notebook (#24183) commit f98db2008a97f4546d036ddf0dddfee8c87eb58a Author: Robert Burke <lostluck@users.noreply.github.com> Date: Tue Nov 15 12:49:23 2022 -0800 Update automation to use Go 1.19 (#24175) Co-authored-by: lostluck <13907733+lostluck@users.noreply.github.com> commit e5f58504eef1fdeebe0402cda8a2df259169c704 Author: Brian Hulette <bhulette@google.com> Date: Tue Nov 15 12:25:13 2022 -0800 Add error reporting for BatchConverter match failure (#24022) * add error reporting for BatchConverters * Test pytorch * Finish up torch tests * yapf * yapf * Remove else commit 3037747f66f0d71d65b6c65745b4f8942c22f05a Author: Danny McCormick <dannymccormick@google.com> Date: Tue Nov 15 14:13:04 2022 -0500 Fix broken notebook (#24179) commit b2b1c739ce37690923891934ee317f799db937a2 Author: MakarkinSAkvelon <67736809+MakarkinSAkvelon@users.noreply.github.com> Date: Tue Nov 15 21:53:06 2022 +0500 [Playground] Move Playground in GKE and Infrastructure change (#23928) * changes to updated master branch * Change workflow * ingress changes * Certificate was added * Updates for cloud build backend * Update main.tf * Create main.tf * Create variables.tf * Update variables.tf * Update main.tf * Update variables.tf * Update main.tf * Create output.tf * Update output.tf * Update output.tf * Update main.tf * Update build.gradle.kts * Update output.tf * Update main.tf * Update main.tf * Update main.tf * Update variables.tf * Update main.tf * Update variables.tf * Update main.tf * Update main.tf * Update main.tf * Update main.tf * Update main.tf * Update main.tf * Update variables.tf * Update main.tf * Update variables.tf * Update main.tf * Update variables.tf * Update main.tf * Update main.tf * Update main.tf * Update main.tf * Update main.tf * Update main.tf * Update main.tf * Update main.tf * Update output.tf * Update main.tf * Update main.tf * Update output.tf * Create variables.tf * Update main.tf * Update main.tf * Delete playground/terraform/infrastructure/cluddns directory * Update main.tf * Update output.tf * Update output.tf * Update build.gradle.kts * Update build.gradle.kts * Update build.gradle.kts * Update build.gradle.kts * Update build.gradle.kts * Update build.gradle.kts * Update README.md * Update README.md * helm folder name was changed * Update README.md * Update build.gradle.kts * Update build.gradle.kts * Update build.gradle.kts * Updates to readme * Fix DNS name * HelmChart was changed * Some workflows were changed * Remove unused file * playground-examples return * add license information * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * remove "stg" folder * Update README.md * Remove unused files * DNS Removed * var name changed * remove DNSName from var file * 1 * Clear terraform * remove unused records * gradle check * grade last change * issue fix * fix * 1 * run * test * Index creation for Gradle * Add IndexCreation in gradle * Update README.md * Update README.md * Fix names for Frontend * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Redis fix * services fix * Update variables.tf * change order in gradle * Fix Config.g.dart file issue * Update README.md * Playground workflow update Co-authored-by: Sergey Makarkin <sergey.makarkin@akvelon.com> Co-authored-by: Sergey Makarkin <sergey_makarkin@quicktest1.c.apache-beam-testing.internal> Co-authored-by: ruslan-ikhsan <ruslan.ikhsanov@akvelon.com> Co-authored-by: Alex Kosolapov <alex.kosolapov@gmail.com> commit 85df5f2eb2f299e28b36be0cce7b9c19d62124da Author: Yi Hu <yathu@google.com> Date: Tue Nov 15 11:38:13 2022 -0500 Eliminate CalciteUtil.CharType logical type (#24013) * Eliminate CalciteUtils.CharType logical type * Replace CalciteUtils.CharType to String Note that CalciteUtils still omits the precision of BINARY/VARBINARY/CHAR/VARCHAR as what it originally did. Support of the precision of these calcite types involves make use of making use of the overload method RelDataTypeFactory.createSqlType(var1, var2). * Replace every reference of CalciteUtil.CharType to generic PassThroughLogicalType check * Add TODO to Support sql types with arguments * Use VariableString in LogicalTypeTestCase commit f349f41010c5b238ff6020f7de718f938eef3c5e Author: alexeyinkin <alexey.inkin@akvelon.com> Date: Tue Nov 15 20:04:01 2022 +0400 Configure flutter_code_editor options with Hugo shortcode (#23926) (#24031) * Configure flutter_code_editor options with Hugo shortcode (#23926) * Minor fixes (#23926) * Refactor after review (#23926) commit 0f4ca6363b3ce0e5de3ad36517bb406aa6391a18 Author: Rebecca Szper <98840847+rszper@users.noreply.github.com> Date: Tue Nov 15 06:10:13 2022 -0800 Editorial review of the ML notebooks. (#24125) * Editorial review of the ML notebooks. * Editorial review of the ML notebooks. * Editorial review of the ML notebooks. * Update examples/notebooks/beam-ml/custom_remote_inference.ipynb Co-authored-by: Danny McCormick <dannymccormick@google.com> * Updating based on feedback * Update examples/notebooks/beam-ml/run_inference_sklearn.ipynb Co-authored-by: Danny McCormick <dannymccormick@google.com> * Update examples/notebooks/beam-ml/run_inference_tensorflow.ipynb Co-authored-by: Danny McCormick <dannymccormick@google.com> * Update examples/notebooks/beam-ml/run_inference_tensorflow.ipynb Co-authored-by: Danny McCormick <dannymccormick@google.com> * Update examples/notebooks/beam-ml/run_inference_tensorflow.ipynb Co-authored-by: Danny McCormick <dannymccormick@google.com> * Updating based on feedback Co-authored-by: Danny McCormick <dannymccormick@google.com> commit 5bd34ede026253326ebff1a7e4f9edb5f71b4a2c Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue Nov 15 07:17:28 2022 -0500 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#24131) Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.3.2 to 1.11.39. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.3.2...feature/s3/manager/v1.11.39) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 2ee809fa0ca7689dd0279e186ebc02d9569a8429 Merge: e3b9bdb2e6 563c66d6fd Author: Alexey Romanenko <33895511+aromanenko-dev@users.noreply.github.com> Date: Tue Nov 15 11:01:14 2022 +0100 Merge pull request #23065: [Website] Update copy icon styles commit e3b9bdb2e607d85a4017ba7839000e92a0ad83c4 Author: Moritz Mack <mmack@talend.com> Date: Tue Nov 15 10:40:50 2022 +0100 [Dockerized Jenkins] Fix build of dockerized jenkins (fixes #24053) (#24054) commit faaac2ab6e010374cb2be0e95a5dd345836a2a2c Author: Moritz Mack <mmack@talend.com> Date: Tue Nov 15 10:38:59 2022 +0100 [Dockerized Jenkins] Update README how to use local repo (#24055) commit 689e70b5131620540faf52e2f1e2dca7a36f269d Author: Damon <damondouglas@users.noreply.github.com> Date: Mon Nov 14 17:34:29 2022 -0800 Implement embedded WebAssembly example (#24081) commit e1bf6c42950e8013f35e35fb9fee8017e01e5010 Merge: eddac84126 10337d2868 Author: Robert Bradshaw <robertwb@gmail.com> Date: Mon Nov 14 15:22:14 2022 -0800 Merge pull request #24160 Rename the test_splits flag to direct_test_splits. commit eddac841261228a2c63fa9b225c520ae0f853806 Author: Pablo <pabloem@users.noreply.github.com> Date: Mon Nov 14 15:05:05 2022 -0800 More dataset templates to clean up (#24162) commit 2adb68bd12743566cc89b596bf204d7c807eb62d Author: Pablo <pabloem@users.noreply.github.com> Date: Mon Nov 14 13:28:13 2022 -0800 Adding a quickstart to README for the TS SDK (#23509) * More of a quickstart for the TS SDK * Update sdks/typescript/README.md Co-authored-by: Danny McCormick <dannymccormick@google.com> * Update sdks/typescript/README.md Co-authored-by: Danny McCormick <dannymccormick@google.com> Co-authored-by: Danny McCormick <dannymccormick@google.com> commit 10337d28685ad5712e2ad8608977ec5c5e0e6b6b Author: Robert Bradshaw <robertwb@gmail.com> Date: Mon Nov 14 12:46:32 2022 -0800 Rename the test_splits flag to direct_test_splits. This avoids possible flag conflicts. commit 48c70cc30742b45b17a1d18ece2f0d079bee3915 Author: arne-alex <108519096+arne-alex@users.noreply.github.com> Date: Mon Nov 14 21:33:02 2022 +0100 Merge pull request #23333: Track time on Cloud Dataflow streaming data reads and export via heartbeats commit 9c83de646ab52bd0b05e3346190dd55cd68b2a8b Author: Johanna Öjeling <51084516+johannaojeling@users.noreply.github.com> Date: Mon Nov 14 21:19:44 2022 +0100 Add more tests for S3 filesystem (#24138) commit 9e9c6d797ba52b460f83131431c8e53aebbbc9ac Merge: d5d76b9745 c600444e1d Author: Ning Kang <ningkang0957@gmail.com> Date: Mon Nov 14 12:06:15 2022 -0800 Merge pull request #24029 from apache/dependabot/npm_and_yarn/sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/loader-utils-1.4.1 Bump loader-utils from 1.4.0 to 1.4.1 in /sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel commit d5d76b974592d45de368ab641647ca5cc4ec12ec Author: Yi Hu <yathu@google.com> Date: Mon Nov 14 15:03:28 2022 -0500 Support SqlTypes Date and Timestamp (MicrosInstant) in AvroUtils (#23969) * Support SqlTypes Date and Timestamp (MicrosInstant) in AvroUtils * Add TODO about java.time migration commit 330cc2010c9f4a2d4e30318bf50a4109ec1cd392 Author: Pablo <pabloem@users.noreply.github.com> Date: Mon Nov 14 12:02:10 2022 -0800 Cleanup stale BQ datasets (#24158) * Cleanup stale BQ datasets * addressing comments commit 4a044999b8ed4bcd41f816f3a23ccb5da00c4c38 Merge: e563b9dd2f 5bd75c25de Author: Heejong Lee <heejong@gmail.com> Date: Mon Nov 14 11:16:00 2022 -0800 Merge pull request #24076 from chamikaramj/multilang_java_updates Updates Multi-lang Java quickstart commit e563b9dd2f3aa0484e6cdc08869991b5e438023e Author: Evgeny Antyshev <eantyshev@gmail.com> Date: Mon Nov 14 20:56:35 2022 +0300 [Tour Of Beam] verify that unit exists when saving progress (#24118) * AIO * Update learning/tour-of-beam/backend/integration_tests/auth_test.go Co-authored-by: Danny McCormick <dannymccormick@google.com> * nit Co-authored-by: Danny McCormick <dannymccormick@google.com> commit 774923e0dd089de870bfa5c77063ae2b28f79347 Merge: 71785de528 1ad0cbc445 Author: Kenn Knowles <kenn@apache.org> Date: Mon Nov 14 09:52:26 2022 -0800 Merge pull request #24141: Fix checkArgument format in GcsPath commit 71785de52864313c2e3b14fe72a2a63281343617 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Nov 14 11:54:37 2022 -0500 Bump github.com/aws/aws-sdk-go-v2/config from 1.17.10 to 1.18.0 in /sdks (#24151) Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.17.10 to 1.18.0. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.17.10...config/v1.18.0) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 50d591d6cb3e799bee4e29dfc593c693a86e6276 Author: Bruno Volpato <bvolpato@google.com> Date: Mon Nov 14 11:50:01 2022 -0500 Change DataflowBatchWorkerHarness doWork error level to INFO (#24135) commit 5a72696bfda09fdb905ba8e58b636f8494ef955f Merge: ee0a5836d6 0633fe9634 Author: Kenn Knowles <kenn@apache.org> Date: Mon Nov 14 08:12:12 2022 -0800 Merge pull request #24149: Remove extraneous jetbrains annotation commit ee0a5836d69b776834eb3bd9b2bd02eb5252c333 Merge: d001a69e1a 137799672e Author: Kenn Knowles <kenn@apache.org> Date: Mon Nov 14 08:11:00 2022 -0800 Merge pull request #24132: Fix checkArgument format string in AvroIO commit d001a69e1a58701d6ed4fcb5e3fb7a0921301dad Author: Yi Hu <yathu@google.com> Date: Mon Nov 14 10:56:54 2022 -0500 Test Dataproc 2.1 with Flink load tests (#24129) * Test Dataproc 2.1 with Flink load tests * Minor fix flink_cluster script commit caabd9be52887ad70c8a4269395c893811ac6a84 Author: Israel Herraiz <ihr@google.com> Date: Mon Nov 14 16:03:39 2022 +0100 Make MonotonicWatermarkEstimator work like its Java SDK equivalent (#24146) * Make MonotonicWatermarkEstimator work like its Java SDK equivalent The current implementation of MonotonicWatermarkEstimator raises an exception with late messages, which makes the watermark estimator barely usable in real world scenarios. This PR fixes #20041 by making this watermark estimator work like its Java SDK equivalent (`WatermarkEstimators.MonotonicallyIncreasing`). * Update unit tests too * Make linter happy commit 451f6b3e7f58d0a3782ad942c6a1fd9f63932024 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Nov 14 09:48:23 2022 -0500 Bump golang.org/x/net from 0.1.0 to 0.2.0 in /sdks (#24153) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.1.0 to 0.2.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](https://github.com/golang/net/compare/v0.1.0...v0.2.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 2bb03d62e2d7dc2d8e39040fc9adebccbde74fde Merge: 4e39ef2041 623083cd0a Author: Alexey Romanenko <33895511+aromanenko-dev@users.noreply.github.com> Date: Mon Nov 14 15:01:13 2022 +0100 Merge pull request #24000: [Website] Change headers size from h4,h3 to h2 commit 563c66d6fd32165da14a07747f2764c17a5d24ea Author: bulat safiullin <bulat.safiullin@akvelon.com> Date: Wed Sep 7 18:28:42 2022 +0600 [Website] update pre tag copy link styles #23064 commit 4e39ef20410ee51c6040317bcd60171e64c5171f Merge: 223768f782 105ed6fedc Author: Alexey Romanenko <33895511+aromanenko-dev@users.noreply.github.com> Date: Mon Nov 14 10:55:33 2022 +0100 Merge pull request #24115: [Website] update go-dependencies.md java-dependencies.md links commit 223768f782f771f0033b8d0686d86cf4c71fad75 Merge: aa0a35dabf a9da2abee6 Author: Kenn Knowles <kenn@apache.org> Date: Sun Nov 13 18:53:13 2022 -0800 Merge pull request #24136: Fix checkArgument format string in ExecutionStateTracker commit 0633fe9634fe61df7cbc0ecac205d81124fd504a Author: Kenneth Knowles <klk@google.com> Date: Sat Nov 12 15:15:16 2022 -0800 Remove extraneous jetbrains annotation commit 83f1bc19b95935e60ca1f4027d4b60c7e738a84a Author: Kenneth Knowles <klk@google.com> Date: Sat Nov 12 14:16:09 2022 -0800 Fix arguments to checkState in BatchViewOverrides commit 1ad0cbc44594d8405bf4b07a126265238013a02a Author: Kenneth Knowles <klk@google.com> Date: Sat Nov 12 13:41:02 2022 -0800 Fix checkArgument format in GcsPath commit aa0a35dabf9c2a0d9822faff06d939d9a77a3ab6 Author: Kenn Knowles <kenn@apache.org> Date: Fri Nov 11 20:26:30 2022 -0800 Fix checkArgument format string in TestStream (#24134) commit a9da2abee6455bc2cf0f18ba5f6cd7bbaeae669f Author: Kenneth Knowles <klk@google.com> Date: Fri Nov 11 16:54:27 2022 -0800 Fix checkArgument format string in ExecutionStateTracker commit 369e2ba8622d3474c14c39b941b2c618842d1e47 Author: Ryan Thompson <ryanthompson591@gmail.com> Date: Fri Nov 11 19:46:07 2022 -0500 Add a ValidatesContainer integration test for use_sibling_sdk_workers (#24099) commit 137799672eb559a7586262e6a8a73d1ab3580e44 Author: Kenneth Knowles <klk@google.com> Date: Fri Nov 11 15:30:01 2022 -0800 Fix checkArgument format string in AvroIO commit 5d2dbf957e4e82fb3980726940df02ac67e563cd Author: Anand Inguva <34158215+AnandInguva@users.noreply.github.com> Date: Fri Nov 11 15:57:28 2022 -0500 Update staging of Python wheels (#24114) Fixes https://github.com/apache/beam/issues/24110 commit c2021bee1eba0322b43c90841397859048296b21 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri Nov 11 15:33:14 2022 -0500 Bump google.golang.org/api from 0.102.0 to 0.103.0 in /sdks (#24049) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.102.0 to 0.103.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.102.0...v0.103.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 6557c91c79480b9d90573d52d257a11c2b160196 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri Nov 11 11:47:12 2022 -0800 Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks (#24112) Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.29.1 to 1.29.2. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.29.1...service/s3/v1.29.2) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 96f9da1ab652156cd143d57e3aa3d94836338f2b Author: Yi Hu <yathu@google.com> Date: Fri Nov 11 14:34:22 2022 -0500 More cleanup containers (#24105) * More cleanup containers * prebuilt_beam_sdk/beam_python_prebuilt_sdk no longer exists in gcr.io Add correct paths to cleanup * Bump grace time to 15 days * Run job daily as in code comment * Set grace period to 30 days commit 836766ddb83d37463e4b036f156b8d7e15e3864b Author: Ritesh Ghorse <riteshghorse@gmail.com> Date: Fri Nov 11 14:30:30 2022 -0500 upgrade testcontainer dependency (#24123) commit 9fcd20c3712536f2d4580beead678cdbb6fd4746 Author: Damon <damondouglas@users.noreply.github.com> Date: Fri Nov 11 11:12:11 2022 -0800 Implement PubsubRowToMessage transform (#23897) * Begin PubsubRowToMessage Impl * Complete working draft * Unit tests validate user and non-user fields * Finish tests on supporting methods * Pass checks before finalizing tests * WIP * fix timestamp * finalize tests * Finalize code comments * Clean up check findings * Add InputSchemaFactory * Patch code comment typo commit 3a6fcc1ca4d07a467464ed1214a94b5c9c147295 Author: Evgeny Antyshev <eantyshev@gmail.com> Date: Fri Nov 11 19:13:45 2022 +0300 disable (#24121) commit 027fb142038b45c443d87af96cac082264c43188 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri Nov 11 10:26:23 2022 -0500 Bump cloud.google.com/go/bigtable from 1.17.0 to 1.18.0 in /sdks (#24113) Bumps [cloud.google.com/go/bigtable](https://github.com/googleapis/google-cloud-go) from 1.17.0 to 1.18.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.17.0...pubsub/v1.18.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/bigtable dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit b2d28a64bb0837dd97e8be7f97f31f88d380f110 Author: Danny McCormick <dannymccormick@google.com> Date: Fri Nov 11 08:45:14 2022 -0500 Add TFX support in pydoc (#23960) * Add TFX support in pydoc * Wording commit 105ed6fedcb6ff33d4a43940c342a653e67cb25e Author: bulat safiullin <bulat.safiullin@akvelon.com> Date: Thu Nov 10 17:47:54 2022 +0600 [Website] update go-dependencies.md java-dependencies.md programming-guide.md links #24084 commit 4b96193250a63b27721a8c5e1a3bd6ecff983093 Author: Brian Hulette <bhulette@google.com> Date: Thu Nov 10 16:51:06 2022 -0800 Remove TheNeuralBit from the pool of Python reviewers (#24108) commit 8f8f089a5f565f179905984cef3522a0577d9219 Author: Sanil Jain <snjain@linkedin.com> Date: Thu Nov 10 16:47:31 2022 -0800 Wire SamzaPipelineOptions to Exeption listener interface (#24109) commit b3186ba91f9bb22764d47b78eb2b7ff017080f75 Author: Ahmed Abualsaud <65791736+ahmedabu98@users.noreply.github.com> Date: Thu Nov 10 18:04:50 2022 -0500 Support using BigQueryIO Storage Read API with SchemaTransforms (#23827) * support schema transform for bq direct read method * use vendor Strings import * add BigQueryServices argument to config object * suppress nullability errors * add package-info.java file to providers subdirectory * removing accidentally created file * added documentation and moved configuration filclass into provider class * validate config params before expansion. config has a validaate me method * updated URN identifier to follow standards commit a4a94da1936d8cd2f8d1145fd2e8329fe06a2990 Author: Oleh Borysevych <oleg.borisevich@akvelon.com> Date: Fri Nov 11 00:28:43 2022 +0200 fixing linter error (#24104) commit 156a6099d71a155c3379340a0b98256306e1755e Author: Danny McCormick <dannymccormick@google.com> Date: Thu Nov 10 17:26:15 2022 -0500 Add blog post on new ML resources (#24071) * Create ml-resources.md * Add ensemble notebook image * Add image link * Fix image link * Add ml-landing page image * Add image + move around * Add paragraph on upcoming changes * Remove bad whitespace commit 38742d40c895f5d38ee6ee5eb328d11b3262307e Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu Nov 10 16:12:40 2022 -0500 Bump github.com/aws/aws-sdk-go-v2/config from 1.5.0 to 1.17.10 in /sdks (#24080) Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.5.0 to 1.17.10. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.5.0...config/v1.17.10) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 92cef32915c40d353b51529a87d8f0131b0a3538 Author: BjornPrime <32173247+BjornPrime@users.noreply.github.com> Date: Thu Nov 10 15:30:37 2022 -0500 Num failed inferences (#23830) * created test_increment_num_failed_inferences and test_num_failed_inferences_no_failures * added assertRaises to test_increment_num_failed_inferences * added num_failed_inferences to _MetricsCollector * changed error handling and update() implementation * updated metric name in tests * removed unnecessary else blocking * removed unnecessary inference_args from test_increment_failed_batches_counter() * changed final test_increment_failed_batches_counter assertion * clarified error handling and updated failed_batches_counter initialization * decreased examples array length to 1 to ensure repeatability * troubleshooting tests * trying to get test_increment_failed_batches_counter to fail as expected * corrected assertion details * simplified assertRaises and added reminder comment to assertEqual counter * lint test * lint test passed, resetting pre-commit-config.yaml * fixed lingering linting issues * shortened comment line to comply with linting * formatter worked its magic commit 2341f61d48d838ddaf2e4fb990a49987244c3513 Author: Evgeny Antyshev <eantyshev@gmail.com> Date: Thu Nov 10 21:24:55 2022 +0300 [Tour Of Beam] handle CORS pre-flight requests (#24083) * cors * README * nit * nit * -headers, -cache * allow-headers * +1h cache,comment commit 36b0c8f1a76865e4dc3b1eaec1566c3ef9fd4345 Author: Evgeny Antyshev <eantyshev@gmail.com> Date: Thu Nov 10 21:21:21 2022 +0300 [Playground] update snippet by persistence_key (#24056) * proto * AIO * -frontend * testify * +license * -build * index * skipKey * Update playground/backend/internal/db/datastore/datastore_db.go Co-authored-by: Danny McCormick <dannymccormick@google.com> Co-authored-by: Danny McCormick <dannymccormick@google.com> commit df553d1a4c6c6cc9b37ada2a6c84c88d8aef38b9 Author: Ahmed Abualsaud <65791736+ahmedabu98@users.noreply.github.com> Date: Thu Nov 10 13:17:36 2022 -0500 Add random string at the end of BigQuery query job name to make it resilient to retries (#24041) * add random string at the end of query job name * use deterministic temp table name commit e439f4120ef4c25aa36e5b03756dc7391bdbd211 Author: Pablo <pabloem@users.noreply.github.com> Date: Thu Nov 10 10:06:26 2022 -0800 Improving stale container cleanup script (#24040) * Improving stale container cleanup script * Avoid also latest image * Update .test-infra/tools/stale_dataflow_prebuilt_image_cleaner.sh Co-authored-by: Yi Hu <huuyyi@gmail.com> * Update .test-infra/tools/stale_dataflow_prebuilt_image_cleaner.sh Co-authored-by: Yi Hu <huuyyi@gmail.com> Co-authored-by: Yi Hu <huuyyi@gmail.com> commit 8d585242e158babd8dd2ca5a6d5a49d353be2935 Author: Anand Inguva <34158215+AnandInguva@users.noreply.github.com> Date: Thu Nov 10 12:58:41 2022 -0500 [Python]Set pickle library at the Pipeline creation stage (#24069) Fixes https://github.com/apache/beam/issues/21615 commit 08b6a524fecc333a12ce42971733ef64ae7d02f1 Author: Janek Bevendorff <janek.bevendorff@uni-weimar.de> Date: Thu Nov 10 18:37:09 2022 +0100 [BEAM-12792] Install pipline dependencies to temporary venv (#16658) commit 74f87b0282a23aa01ecb63288013311d207960aa Merge: 38a85b288a 383ea77bc9 Author: Robert Bradshaw <robertwb@gmail.com> Date: Thu Nov 10 09:10:37 2022 -0800 Merge pull request #23985 Support dynamic sharding in the worker. commit 38a85b288ad10171c444d4092eb969ca5de58af3 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu Nov 10 10:39:15 2022 -0500 Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks (#24077) Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.11.1 to 1.29.1. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.11.1...service/s3/v1.29.1) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit c5110bae06fb9db6cace1f7657745fa97e3f58af Author: Yi Hu <yathu@google.com> Date: Thu Nov 10 10:33:55 2022 -0500 Fix FhirIO javadoc format broken (#24072) commit 623083cd0a72d86c7ecba7653e9a8564861abc4d Author: bulat safiullin <bulat.safiullin@akvelon.com> Date: Mon Nov 7 16:08:39 2022 +0600 [Website] change headers size from h4,h3 to h2 #24082 commit 5bd75c25de291e517cc5c5799ae4adaaaaceacb7 Author: Chamikara Jayalath <chamikaramj@gmail.com> Date: Wed Nov 9 18:04:28 2022 -0800 Updates Multi-lang Java quickstart commit 4522f4ce916223afb3c4edd02a7be6018d0ce5a4 Author: Trevor Gevers <tlgeverssa@gmail.com> Date: Wed Nov 9 18:56:18 2022 -0600 Update datastore_wordcount.py (#23724) commit 96cc252348ce7229731be3c74bf5848d1b004c08 Author: tvalentyn <tvalentyn@users.noreply.github.com> Date: Wed Nov 9 16:08:00 2022 -0800 Remove a duplicate label (#24043) commit 62a6bd1e9317a7658810e9a0eebfe925460e8982 Author: Thiago Nunes <thiagotnunes@google.com> Date: Thu Nov 10 08:45:27 2022 +1100 test: add more tests to throughput estimator (#23915) Verifies that getting the throughput when no updates have occurred for the size of the window should return 0. commit e6647c34bb367b3058c42d0201150395c534e7ce Author: Ritesh Ghorse <riteshghorse@gmail.com> Date: Wed Nov 9 15:12:59 2022 -0500 skip output coder field in exp request (#24066) commit 383ea77bc982416afcbbbe11a4dc1f1424670589 Author: Robert Bradshaw <robertwb@gmail.com> Date: Wed Nov 9 10:49:37 2022 -0800 Reduce flakiness of time-based split manager test. Increase wait time from a tenth to half a second. commit 539fa9159ffc116b2e79e6de2804dfdd1c1e4722 Author: Danny McCormick <dannymccormick@google.com> Date: Wed Nov 9 11:45:20 2022 -0500 Convert initialisms to all caps (#24061) * Convert initialisms to all caps * Fix test refs commit 4941b9633c6ccf9eb8aad554186aac24ca7b7492 Author: Ryan Thompson <ryanthompson591@gmail.com> Date: Wed Nov 9 11:25:48 2022 -0500 added comments for tensorflow notebook (#23726) * added comments for tensorflow notebook * added note to string type commit 858f5048f578548a63e5e4319a7ab5363311c269 Merge: 95c121b254 1929968891 Author: Alexey Romanenko <33895511+aromanenko-dev@users.noreply.github.com> Date: Wed Nov 9 16:35:19 2022 +0100 Merge pull request #24057: [Spark Dataset runner] Enable projection pushdown for Spark dataset runner commit 95c121b2549d3f8fda7668e3759b8991b55d9865 Author: Oleh Borysevych <oleg.borisevich@akvelon.com> Date: Wed Nov 9 16:03:33 2022 +0200 Fix dependency mismatch in Playground Java runner (#24059) * fixing existing and potential dependency mismatch * extract grpc version commit 1929968891d629c83249c574c28db6c103ba9271 Author: Moritz Mack <mmack@talend.com> Date: Wed Nov 9 13:35:12 2022 +0100 [Spark Dataset runner] Enable projection pushdown for Spark dataset runner. commit 63362f5ba60a22e77e54c2df47a844f031036309 Author: Johanna Öjeling <51084516+johannaojeling@users.noreply.github.com> Date: Wed Nov 9 07:49:54 2022 +0100 [Go SDK] S3 implementation of the Beam filesystem (#23992) * Implement filesystem for S3 * Update CHANGES.md commit 73142bad2472e4516831c24a8dfb05c9acebf791 Author: Kanishk Karanawat <kkdoon@gmail.com> Date: Tue Nov 8 20:46:04 2022 -0500 Handle Avro schema generation for logical data types in BigQueryAvroUtils (#23620): handle avro logical types for TableFieldSchema to Avro schema conversion Co-authored-by: Kanishk Karanawat <kkaranawat@twitter.com> commit 8db8fae173056c0d89a36a8a5ce0443b6b9e54fa Author: Robert Burke <lostluck@users.noreply.github.com> Date: Tue Nov 8 15:56:53 2022 -0800 [Go] Add pipeline resource hints to CHANGES.md (#24036) commit a84b7dc179fed5dfc1d48dddfb04266de3716e31 Author: Robert Bradshaw <robertwb@gmail.com> Date: Mon Nov 7 16:35:25 2022 -0800 Make mypy happy. commit b9655e7de1a682d8ec4efcafb4d610f794e1b40e Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue Nov 8 12:27:38 2022 -0800 Bump cloud.google.com/go/storage from 1.27.0 to 1.28.0 in /sdks (#24028) Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.27.0 to 1.28.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/spanner/v1.27.0...spanner/v1.28.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/storage dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 708e05f0dc6c0a2e62674cd0ecb8a6744ae57d3a Author: Brian Hulette <bhulette@google.com> Date: Tue Nov 8 12:18:23 2022 -0800 Retroactively announce Batched DoFn support in 2.42.0 Blog (#24011) * Retroactively announce Batched DoFn support in 2.42.0 * Add to blog as well commit b9ff75a18b40141a8d2340e397fd70382b3d4e8f Author: Israel Herraiz <ihr@google.com> Date: Tue Nov 8 20:03:43 2022 +0100 Update my Twitter handle (#23653) commit bf621ccb88023e0bc09752978c099ac4629086e2 Author: Rebecca Szper <98840847+rszper@users.noreply.github.com> Date: Tue Nov 8 10:50:02 2022 -0800 Editorial review of the ML base API descriptions (#24026) commit f0a6ff46619ee1fd0639fa3f06e04708f69dee74 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue Nov 8 13:04:03 2022 -0500 Bump cloud.google.com/go/bigtable from 1.16.0 to 1.17.0 in /sdks (#24027) Bumps [cloud.google.com/go/bigtable](https://github.com/googleapis/google-cloud-go) from 1.16.0 to 1.17.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.16.0...pubsub/v1.17.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/bigtable dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 7dac3f5ef40b5d24b24d9ce5bec4717284260b85 Author: Lukasz Cwik <lcwik@google.com> Date: Tue Nov 8 09:42:29 2022 -0800 [#21250] Trivial removal of loop over something that always has one element (#24014) Multiplexing was put into the PCollectionConsumerRegistry a long time ago and this seems to have been missed during that migration. commit a6a9b23fd5b74142c35948f10d1840b882817246 Author: Robert Burke <lostluck@users.noreply.github.com> Date: Tue Nov 8 09:19:49 2022 -0800 [Go] Pipeline Resource Hints (#23990) commit 8cf2a63cc25399109f64b84ddfcd7c74d21e61ce Author: tvalentyn <tvalentyn@users.noreply.github.com> Date: Tue Nov 8 09:19:16 2022 -0800 Update release notes. (#23986) commit c600444e1dfa2f4afaad056bc65fa009a60a32bb Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue Nov 8 06:22:14 2022 +0000 Bump loader-utils Bumps [loader-utils](https://github.com/webpack/loader-utils) from 1.4.0 to 1.4.1. - [Release notes](https://github.com/webpack/loader-utils/releases) - [Changelog](https://github.com/webpack/loader-utils/blob/v1.4.1/CHANGELOG.md) - [Commits](https://github.com/webpack/loader-utils/compare/v1.4.0...v1.4.1) --- updated-dependencies: - dependency-name: loader-utils dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> commit 62d8bdc604dfeccae08d00225bce8c9b9c3ffb28 Merge: d4142f7eab 930c4d2d87 Author: Robert Bradshaw <robertwb@gmail.com> Date: Mon Nov 7 15:31:37 2022 -0800 Merge pull request #23789 Better surfacing of Scala support via Scio. commit 7f7713de09b3b0e400ed02ffdfcb34a6a8b6e1e4 Merge: 5f587b9339 d4142f7eab Author: Robert Bradshaw <robertwb@gmail.com> Date: Mon Nov 7 15:29:27 2022 -0800 Merge branch 'master' into javascript-liquid-sharding commit 5f587b9339b3b19ae5c5b1cf15e3ce218f75d21a Author: Robert Bradshaw <robertwb@gmail.com> Date: Mon Nov 7 15:25:33 2022 -0800 Clarifying comments. commit 60fe3cdddf9d3c27d9493b345d1f5f2a9284a76f Author: Robert Bradshaw <robertwb@gmail.com> Date: Mon Nov 7 15:06:26 2022 -0800 Update style sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py Co-authored-by: Danny McCormick <dannymccormick@google.com> commit d4142f7eab7d5a3d21a3d3930c448fe403880f93 Author: Robert Burke <lostluck@users.noreply.github.com> Date: Mon Nov 7 12:11:21 2022 -0800 Switch && for || to fix bug in #23889 resolution (#24017) commit 45100d708d954b25b5a884dc3c7a965f008228c6 Author: Danny McCormick <dannymccormick@google.com> Date: Mon Nov 7 14:53:24 2022 -0500 Add files then check cached diff to get untracked files commit 380d4730d3c52b5c2165bdb84aac11d0feccfe4f Author: scwhittle <scwhittle@users.noreply.github.com> Date: Mon Nov 7 19:33:37 2022 +0100 Enforce splitting invariants by ensuring split state is reset in the same synchronized block as window index increments. (#23882) * Enforce splitting invariants by ensuring split state is reset in the same synchronized block as window index increments. Fixes #23881. * Add missing currentElement = null; Co-authored-by: Lukasz Cwik <lcwik@google.com> commit 8b3fd2e0d652045bf94b83638129bb12562ce738 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Nov 7 10:33:26 2022 -0800 Bump google.golang.org/api from 0.101.0 to 0.102.0 in /sdks (#23957) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.101.0 to 0.102.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.101.0...v0.102.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 9533fc3757d3041135cbe2f5078dff0855adc1f1 Author: capthiron <24925821+capthiron@users.noreply.github.com> Date: Mon Nov 7 19:32:18 2022 +0100 feat: implement bigtable io connector with write capabilities (#23411) commit c4218e572aa8a8f46b433b60073a59fdd7fdf148 Merge: 23676a9ec1 a5ee669a43 Author: Alexey Romanenko <33895511+aromanenko-dev@users.noreply.github.com> Date: Mon Nov 7 18:24:45 2022 +0100 Merge pull request #23934: [Website] change case studies page mobile layout commit 23676a9ec17069b309e8803d1fbfeb8009035828 Author: Oleh Borysevych <oleg.borisevich@akvelon.com> Date: Mon Nov 7 17:24:59 2022 +0200 [Tour of Beam] Learning content for "Introduction" module (#23085) * learning content for introduction module * removing white spaces from md files * delete whitespaces * delete whitespaces in python * delete whitespace #2 * divide pipeline concepts * add pipeline example concepts * adding category tag to python examples * adding category to java examples * adding category to go examples * fixed go example * fixed go example compilation * fixing python duplicate example names * add runner concepts * fixing java examples * add licence for runner unit * some minor fixes for unit names * fixed unit name * resolving CR comments * adding complexity to examples * adding tags * fixed go example compilation * fixed python example with duplicate transform * change indent python * fixing missing pipeline options * change arrow symbol * delete example prefix * minor formatting and readability fixes * add example description * minor fix * minor code review comment Co-authored-by: Abzal Tuganbay <abzal.tugan@gmail.com> commit 98d2ffd69ab9d5c49196a78f7c25daacb57b6478 Author: Brian Hulette <bhulette@google.com> Date: Mon Nov 7 07:03:12 2022 -0800 Make `documentation/io/connectors/` canonical (#23877) * Drop documentation/io/built-in/ * Update links in Custom I/O pattern commit c0b41fa045418beb782c0a59e6f5c511da898c53 Author: Danny McCormick <dannymccormick@google.com> Date: Mon Nov 7 09:29:08 2022 -0500 Use git diff instead of git diff-index to avoid file timestamp changes being picked up commit 0476d6498217b7b0e833643c47fef2c4bdaec529 Merge: 1d5fc14031 6ae37b6f82 Author: Alexey Romanenko <33895511+aromanenko-dev@users.noreply.github.com> Date: Mon Nov 7 15:28:59 2022 +0100 Merge pull request #23962: [CdapIO] Add sparkreceiver:2 module. commit 1d5fc14031d583e92018ebfbeaff9b4e3b978fdb Author: Danny McCormick <dannymccormick@google.com> Date: Mon Nov 7 09:14:24 2022 -0500 Remove quiet flag on debug commit c08273b9355566181d2d5c0792130f8d5cc1428c Author: Danny McCormick <dannymccormick@google.com> Date: Mon Nov 7 09:12:26 2022 -0500 Correctly print diff and swallow empty commits for the moment commit 66fb431deb77db5f7f0fac31e6bf3b2c094c8ac5 Author: Danny McCormick <dannymccormick@google.com> Date: Mon Nov 7 09:08:53 2022 -0500 Print diff and scope to state path commit e0e10b9e5432643b884c381d145e5924cc4ef193 Author: Danny McCormick <dannymccormick@google.com> Date: Sun Nov 6 12:12:22 2022 -0500 PR Bot - Dont throw error on return code 1 commit bbcb790461c47d22399c681cd42c434b206eb550 Author: Danny McCormick <dannymccormick@google.com> Date: Sun Nov 6 11:59:32 2022 -0500 Fix pr bot - exec doesn't allow command chaining commit 88f9a66a9f4cc2fe3259d5ebd83ebb472a29dd5e Author: Danny McCormick <dannymccormick@google.com> Date: Sun Nov 6 11:46:34 2022 -0500 Fix diff to stop repeated bot runs commit ddb4f2d53538fea68383fe43221d4074d05d11b9 Author: bullet03 <bulat.safiullin@akvelon.com> Date: Fri Nov 4 16:11:10 2022 -0700 Website add and update logos (#23899) * [Website] add and update logos * Delete talend.svg:Zone.Identifier commit 6d42219ff547720996489aa29de0d64246db2541 Author: Robert Bradshaw <robertwb@gmail.com> Date: Fri Nov 4 14:30:18 2022 -0700 Enable more portable-runner requiring tests. (#23970) * Enable more portable-runner requiring tests. * Run all basic tests on portable runner as well. * Mark ULR tests. commit 1a643d16112a5f3f1418f1be393de51af65d752b Author: David Cavazos <dcavazos@google.com> Date: Fri Nov 4 13:45:12 2022 -0700 Beam starter projects blog post (#23964) Co-authored-by: Ahmet Altay <aaltay@gmail.com> commit d10b4a28fff667289c146152535275035f2dedf7 Author: Ryan Thompson <ryanthompson591@gmail.com> Date: Fri Nov 4 16:06:49 2022 -0400 removed trailing whitespace (#23987) commit 3bd697561aab4a5d2aa49d29143695d6d57a834a Author: Robert Bradshaw <robertwb@gmail.com> Date: Fri Nov 4 11:43:47 2022 -0700 Add dynamic splitting support to the worker. commit ed520243f472c7224a1f8c74a0c2e8f81965cbfc Author: Robert Bradshaw <robertwb@gmail.com> Date: Fri Nov 4 11:07:40 2022 -0700 Add the a Reshuffle operation and use it in Create. This allows created sources to be staticly and dynamically split. commit 9283512544b8de169be9cd7795b42284fdf45b87 Author: Robert Bradshaw <robertwb@gmail.com> Date: Fri Nov 4 10:28:51 2022 -0700 Add the ability to schedule splits on the ULR via a pipeline option. This can be used to test arbitrary SDKs for behavioral correctness for dynamic splitting. These splits are wall-time based, which isn't ideal, but easier than setting up and managing a cross-process synchronization channel. commit ed7b560d7ce7dda0b55a067b9d9a1a9d692a7d7a Author: Robert Bradshaw <robertwb@gmail.com> Date: Fri Nov 4 09:46:19 2022 -0700 Compute element counts for all PCollections. commit 1cfdb127983a55c77b3259c4fdd248497b28e250 Author: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Date: Fri Nov 4 15:43:16 2022 -0400 Add custom inference fn suport to the sklearn model handlers (#23642) * Add custom inference fn suport to the sklearn model handlers * Clean up import order * Update typing, add numpy unit test * Add Pandas unit test * Formatting, linting * yapf run * Remove trailing whitespace * import order * Change inference_fn to keyword-only arg commit 8617b86fd792ac555233dd79c8ede34c2ce67ecf Merge: aa178f8252 2496a0d1e0 Author: Robert Bradshaw <robertwb@gmail.com> Date: Fri Nov 4 11:42:21 2022 -0700 Merge pull request #23978 Add basic counter support to the typescript SDK. commit 2496a0d1e0fda74e8fc0f24eee011362426369a2 Merge: 64abcbe4a3 aa178f8252 Author: Robert Bradshaw <robertwb@gmail.com> Date: Fri Nov 4 11:15:47 2022 -0700 Merge branch 'master' into javascript-metrics commit aa178f825223cbb0a6208fad0e02f98292eeded1 Merge: 97628802fe 1807c307aa Author: Robert Bradshaw <robertwb@gmail.com> Date: Fri Nov 4 09:34:11 2022 -0700 Merge pull request #23976 Correctly capture log levels and attach stage information. commit 64abcbe4a3a15160aba3f3163ff8cbf5186bbfdb Author: Robert Bradshaw <robertwb@gmail.com> Date: Fri Nov 4 09:27:22 2022 -0700 Only report counters that were actually used. commit 1807c307aa1e493acb2d8ba542baffefcb96e7e2 Author: Robert Bradshaw <robertwb@gmail.com> Date: Fri Nov 4 08:52:21 2022 -0700 Remove obsolete TODO. commit 97628802fe4cf6640a3b6854e27b410dd5731379 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri Nov 4 11:47:07 2022 -0400 Bump cloud.google.com/go/datastore from 1.8.0 to 1.9.0 in /sdks (#23916) Bumps [cloud.google.com/go/datastore](https://github.com/googleapis/google-cloud-go) from 1.8.0 to 1.9.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/documentai/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/asset/v1.8.0...asset/v1.9.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/datastore dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 058c56af735e10184d1501cc08cd16af4d21e64a Author: Ayush Sharma <114604338+ayushthe1@users.noreply.github.com> Date: Fri Nov 4 20:58:11 2022 +0530 [Task]: PR Bot will push commits only if they are non-empty (#23937) * fix pr-bot * fix the git pr bot commit 7dba78d611089a1826c5655ed7ffcdf3f6007eae Author: Danny McCormick <dannymccormick@google.com> Date: Fri Nov 4 09:35:16 2022 -0400 Immediately truncate full restriction on drain of periodic impulse (#23765) * Immediately truncate full restriction on drain of periodic impulse * Whitespace format commit 16852518861514a56ad1b57113e3815cfcc8533a Author: Anand Inguva <34158215+AnandInguva@users.noreply.github.com> Date: Fri Nov 4 08:39:34 2022 -0400 TFX image classification example (#23456) * TFX image classification example * TFX image classification with serving model * Add TF Model Wrapper * Clean up code * Refactoring * Add a test for tfx_bsl RunInference * fixup lint * Refactoring * Fixup lint * Add brief summary for the example * Apply suggestions from code review Co-authored-by: Andy Ye <andyye333@gmail.com> * Refactoring code and add comments * Update help description * reorder imports * Reorder imports again * Add docstring * Refactoring * Add pillow to tfx pipeline requirements * Move inferencePostCommitIT to Python 3.9 suite * Uncomment other postcommit suites Co-authored-by: Andy Ye <andyye333@gmail.com> commit 30b26171b4038c1c976adbc3be386e118b35153b Author: Danny McCormick <dannymccormick@google.com> Date: Fri Nov 4 08:22:17 2022 -0400 Update watermark during periodic sequence/impulse (#23507) * Update watermark during periodic sequence/impulse * Remove extraneous import * Formatting * Linting * Only run on dataflow for guaranteed watermark support * More permissive test to avoid timing issues * Test pipeline options * Fix test * Formatting * Formatting * Apply feedback - cleanup/naming/flink * Format * Unused import commit 1563b9721b4bee5a22d384b282762726247e0703 Author: Brian Hulette <bhulette@google.com> Date: Fri Nov 4 05:09:39 2022 -0700 Improve Iterator error message (#23972) commit e3987b5435d39c5fb058d4085eb5e210f8756224 Author: Robert Bradshaw <robertwb@gmail.com> Date: Thu Nov 3 17:39:49 2022 -0700 old prettier change commit 89c8dd5ca23591be6c2778045df32c7d08d9f99e Author: Robert Bradshaw <robertwb@gmail.com> Date: Thu Nov 3 17:12:26 2022 -0700 Add distribution metric type. commit e4e606a8f2652c2fa761a95462f230385b6bcb47 Author: Robert Bradshaw <robertwb@gmail.com> Date: Thu Nov 3 17:09:03 2022 -0700 Support metrics over the portability API. commit e04f3d6fdb39528264cb2a14e60e565991121f1c Author: Robert Bradshaw <robertwb@gmail.com> Date: Thu Nov 3 16:52:51 2022 -0700 Add basic counter setting and getting to the typescript SDK. commit 1daa9c373ac7fbdb696120eeaa2ca9e85a679a1f Author: Robert Bradshaw <robertwb@gmail.…

refactor: resolve some warnings

* remove fastjson library * declare jackson databind artifact * resolve style issue * resolve style issue#2 * resolve unusedDeclaredArtifacts issue * resolve unusedDeclaredArtifacts issue#2 * spotless apply * 1. add ALLOW_UNQUOTED_FIELD_NAMES to object mapper 2. rollback changes of tests * rollback changes of tests * rollback changes of tests #2 * rollback package-lock.json * add breaking change comment related to this PR * rollback JAVA_DEPENDENCY_OWNERS.yaml since it's removed in master branch * add ALLOW_TRAILING_COMMA to tolerant trailing comma * Update sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/TableUtils.java more configs to default behaviour Co-authored-by: Yi Hu <huuyyi@gmail.com> * fix syntax error * make object mapper to be final --------- Co-authored-by: Yi Hu <huuyyi@gmail.com>

…ub Actions (#28390) * # This is a combination of 2 commits. # This is the 1st commit message: added beam_PostCommit_Java_ValidatesRunner jobs to GitHub Actions # This is the commit message #2: Don't improperly filter newly-added elements that overlap with a delete. * merge master into beam_PostCommit_Java_ValidatesRunner branch * updated README file

* updated CHANGELOG.md [ci skip] * updated setup.py [ci skip]

davorbonaci added 2 commits February 29, 2016 10:58

Delete windmill.proto (not part of Beam)

6fe7e94

Delete windmill_service.proto (not part of Beam)

681ca46

asfgit closed this in c9e5525 Mar 1, 2016

aljoscha referenced this pull request in aljoscha/beam May 10, 2016

Merge pull request #2 from aljoscha/fix-flink-tests

6fec570

Fix more failing tests on Flink

silviulica mentioned this pull request Jul 7, 2016

[BEAM-378] integrate setuptools in Maven build #537

Closed

4 tasks

charlesccychen pushed a commit to cosmoskitten/beam that referenced this pull request Jul 25, 2016

Merge pull request apache#2 from chamikaramj/labels

cc5feaa

Fixes examples

cosmoskitten pushed a commit to cosmoskitten/beam that referenced this pull request Sep 6, 2016

Merge pull request apache#2 from bjchambers/splittable-do-fn

788b7db

DoFnInvokers work for Splittable DoFn.

charlesccychen pushed a commit to cosmoskitten/beam that referenced this pull request Oct 14, 2016

Merge pull request apache#2 from apache/python-sdk

e3d6095

Syncing from original

cosmoskitten pushed a commit to cosmoskitten/beam that referenced this pull request Mar 21, 2017

Merge pull request apache#2 from aljoscha/finish-pr-2263-flink-passer…

6aba015

…t-check [BEAM-1726] Obey watermark hold in DoFnOperato.processWatermark2()

cosmoskitten pushed a commit to cosmoskitten/beam that referenced this pull request Jun 7, 2017

This closes apache#2.

9c5c0c0

xumingming mentioned this pull request Jun 17, 2017

[BEAM-2446] restrict scope of BeamSqlEnv in dsl query #3372

Closed

lukecwik mentioned this pull request Aug 15, 2017

[BEAM-2728] Extension for sketch-based statistics #3686

Closed

6 tasks

lukecwik referenced this pull request in lukecwik/incubator-beam Mar 12, 2018

Merge pull request #2 from lukecwik/flink_wordcap

1d8a48d

Add pipeline options to the Python ULR prepare job request.

echauchot mentioned this pull request Apr 4, 2018

[BEAM-2852] Add support for Kafka as source/sink on Nexmark #5019

Merged

10 tasks

charlesccychen pushed a commit to cosmoskitten/beam that referenced this pull request Apr 4, 2018

Merge pull request apache#2 from echauchot/fix_pr_5019

9d73f3a

Fix the code that uses Long key whereas there is no key in the PCollection

charlesccychen pushed a commit to cosmoskitten/beam that referenced this pull request Apr 12, 2018

[BEAM-3942] Applied reviewers' suggestions apache#2

6cfd75b

mareksimunek referenced this pull request in mareksimunek/beam May 9, 2018

Merge pull request #2 from seznam/pete/clean-flink-source-exit

227403c

#! [euphoria-kafka] Properly quit read loop when interrupted

charlesccychen pushed a commit to cosmoskitten/beam that referenced this pull request Jun 21, 2018

Merge pull request apache#2 from swegner/configure_lint

318b78e

Add project configuration for lint warnings and disable dep-ann in SQL.

kohlerm mentioned this pull request Aug 31, 2018

[BEAM-313] Enable the use of an existing spark context with the SparkPipelineRunner #401

Merged

charlesccychen pushed a commit to cosmoskitten/beam that referenced this pull request Sep 16, 2018

Merge pull request apache#2 from lukecwik/pr6328

408473d

[BEAM-4176] Add the ability to allow for runners to register native transforms.

charlesccychen pushed a commit to cosmoskitten/beam that referenced this pull request Sep 29, 2018

Merge pull request apache#2 from swegner/website_branch_fail

67fe0a9

fixup! Simplify website publishing logic

charlesccychen pushed a commit to cosmoskitten/beam that referenced this pull request Oct 1, 2018

[BEAM-4496] Fix apache#2 branch fetch failure on job_PostCommit_Websi…

8935b74

…te_Publish

charlesccychen pushed a commit to cosmoskitten/beam that referenced this pull request Oct 1, 2018

[BEAM-4496] Fix apache#2 branch fetch failure on job_PostCommit_Websi…

d9ca6e3

…te_Publish

charlesccychen pushed a commit to cosmoskitten/beam that referenced this pull request Oct 1, 2018

[BEAM-4496] Fix apache#2 branch fetch failure on job_PostCommit_Websi…

616e76e

…te_Publish

charlesccychen pushed a commit to cosmoskitten/beam that referenced this pull request Oct 1, 2018

[BEAM-4496] Fix apache#2 branch fetch failure on job_PostCommit_Websi…

bbdff1b

…te_Publish

charlesccychen pushed a commit to cosmoskitten/beam that referenced this pull request Oct 1, 2018

[BEAM-4496] Fix apache#2 branch fetch failure on job_PostCommit_Websi…

035b53d

…te_Publish

swegner pushed a commit that referenced this pull request Oct 2, 2018

[BEAM-4496] Fix #2 for branch fetch failure on job_PostCommit_Website…

853758c

…_Publish (#6531)

charlesccychen pushed a commit to cosmoskitten/beam that referenced this pull request Oct 29, 2018

address review feedback apache#2

0b1ad41

PawasChhokra pushed a commit to PawasChhokra/beam that referenced this pull request Aug 27, 2021

Merge pull request apache#2 from linkedin/update-li-trunk

bef03e2

Update li_trunk branch to no longer use internal samza-versions

thiagotnunes mentioned this pull request Jan 14, 2022

[BEAM-12164]: Add SDF for reading change stream records #16514

Merged

rohdesamuel mentioned this pull request Jan 19, 2022

[BEAM-13685] Enable users to specify cache directory under Interactive Beam options #16555

Merged

4 tasks

tvalentyn mentioned this pull request Mar 17, 2022

[BEAM-13314]Revise recommendations to manage Python pipeline dependencies. #16938

Merged

4 tasks

robertwb added a commit that referenced this pull request May 6, 2022

Merge pull request #2 from pabloem/core-api

4d16093

sketching the core API for JS SDK

kennknowles mentioned this pull request Jun 3, 2022

migrate beam_it_args -> beam_it_options #18434

Closed

damccorm mentioned this pull request Jun 4, 2022

Stateful DoFn with Python sdk and DataFlow runner #20520

Open

danthev mentioned this pull request Jul 12, 2022

[BEAM-22089] Fix query retry in Java FirestoreIO #22175

Merged

ryanthompson591 mentioned this pull request Oct 19, 2022

Move Tensorflow Documentation #23729

Merged

4 tasks

ahmedabu98 mentioned this pull request Nov 2, 2022

[Bug]: Number of Python examples are failing for Flink and Spark on 2.43.0 release branch #23907

Closed

xinyuiscool mentioned this pull request Dec 2, 2022

Samza runner support for non unique stateId across multiple ParDos #24276

Merged

4 tasks

sjvanrossum pushed a commit to sjvanrossum/beam that referenced this pull request May 17, 2023

Merge pull request apache#2 from laysakura/refactor/rm-warnings

4bfdd26

refactor: resolve some warnings

gabrywu added a commit to gabrywu/beam that referenced this pull request Aug 16, 2023

rollback changes of tests apache#2

d6acac1

pl04351820 pushed a commit to pl04351820/beam that referenced this pull request Dec 20, 2023

chore: release 1.6.2 (apache#2)

3a37ce9

* updated CHANGELOG.md [ci skip] * updated setup.py [ci skip]

JayajP added a commit to JayajP/beam that referenced this pull request Dec 27, 2023

test commit apache#2

d50be23

rajkgupt mentioned this pull request Apr 22, 2024

fix examples on website #31066

Merged

3 tasks

t2h6 mentioned this pull request Jul 3, 2024

Address a data race on numMessages in DirectStreamObserver.java #31763

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-1] Removal of a few windmill-related protos that aren't Beam-specific #2

[BEAM-1] Removal of a few windmill-related protos that aren't Beam-specific #2

davorbonaci commented Feb 29, 2016

kennknowles commented Mar 1, 2016

[BEAM-1] Removal of a few windmill-related protos that aren't Beam-specific #2

[BEAM-1] Removal of a few windmill-related protos that aren't Beam-specific #2

Conversation

davorbonaci commented Feb 29, 2016

kennknowles commented Mar 1, 2016