Cloud Spanner DML & PartitionedDML support #3703

snehashah16 · 2018-09-19T17:53:53Z

No description provided.

vam-google · 2018-09-19T19:38:54Z

@snehashah16 please provide detailed description (I don't have context on this).

snehashah16 · 2018-09-19T20:25:42Z

@snehashah16 please provide detailed description (I don't have context on this).

@vam-google this is the PR for launching DML & PDML in Cloud Spanner. I would like to have this released in this week's java release if possible. I have an email chain ongoing with you, jesse & jeff (and garrett). This merge will be succesful when the protos have been merged into google-api-grpc (and generated code created).

snehashah16 · 2018-09-21T18:53:43Z

Spoke to @vam-google offline and this will be merged after the release this week.

vam-google

In general looks good with several (mostly minor) comments.

...loud-clients/google-cloud-spanner/src/main/java/com/google/cloud/spanner/DatabaseClient.java

google-cloud-clients/google-cloud-spanner/src/main/java/com/google/cloud/spanner/ResultSet.java

-   * UnsupportedOperationException}. This method must be called after {@link #next()} has
-   * returned @{code false}. Calling it before that will result in an {@code IllegalStateException}.
+   * com.google.cloud.spanner.ReadContext.QueryAnalyzeMode)} method or for DML statements in
+   * {@link ReadContext#executeQuery(Statement, QueryOption)}. Attempts to call this method on


...e-cloud-clients/google-cloud-spanner/src/main/java/com/google/cloud/spanner/SpannerImpl.java

@@ -1334,6 +1344,83 @@ private void backoff(Context context, BackOff backoff) {
    }
  }

+  static class PartitionedDMLTransaction implements SessionTransaction {
+    private ByteString transactionId;


...e-cloud-clients/google-cloud-spanner/src/main/java/com/google/cloud/spanner/SpannerImpl.java

@@ -1467,6 +1559,12 @@ void rollback() {
      }
    }

+    long getSeqNo() {
+      synchronized (lock) {


...-cloud-clients/google-cloud-spanner/src/test/java/com/google/cloud/spanner/it/ITDMLTest.java

+
+    long rowCount = client.executePartitionedUpdate(Statement.of(UPDATE_DML));
+    // Note: With PDML there is a possibility of network replay or partial update to occur, causing
+    // this assert to fail. We should remove this assert if it is a recurring failure in IT tests.


...-cloud-clients/google-cloud-spanner/src/test/java/com/google/cloud/spanner/it/ITDMLTest.java

+/** Integration tests for DML. */
+@Category(IntegrationTest.class)
+@RunWith(JUnit4.class)
+public final class ITDMLTest {


snehashah16

PTAL

vam-google

LGTM, to not block the upcoming release, but I still highly dislike DML in ReadContext (please see the new comment).

google-cloud-clients/google-cloud-spanner/src/main/java/com/google/cloud/spanner/ResultSet.java

-   * UnsupportedOperationException}. This method must be called after {@link #next()} has
-   * returned @{code false}. Calling it before that will result in an {@code IllegalStateException}.
+   * com.google.cloud.spanner.ReadContext.QueryAnalyzeMode)} method or for DML statements in
+   * {@link ReadContext#executeQuery(Statement, QueryOption)}. Attempts to call this method on


...-cloud-clients/google-cloud-spanner/src/test/java/com/google/cloud/spanner/it/ITDMLTest.java

+/** Integration tests for DML. */
+@Category(IntegrationTest.class)
+@RunWith(JUnit4.class)
+public final class ITDMLTest {


This reverts commit c163907.

…s#3703)" (googleapis#3741)" This reverts commit 3c9bbf2.

…3741)" (#3755) This reverts commit 3c9bbf2.

…oogleapis#3703)" (googleapis#3741)" (googleapis#3755)" This reverts commit 65ca2f3.

…3703)" (#3741)" (#3755)" (#3761) This reverts commit 65ca2f3.

Revert "Revert "Revert "Revert "Cloud Spanner DML & PartitionedDML support (#3703)" (#3741)" (#3755)" (#3761)" This reverts commit 68f38e1.

* Launch Cloud Spanner DML & PartitionedDML support Revert "Revert "Revert "Revert "Cloud Spanner DML & PartitionedDML support (#3703)" (#3741)" (#3755)" (#3761)" This reverts commit 68f38e1. * Cloud Spanner - Bug fix for seqNo, fix javadoc errors

- b2056bd147466fec8a817885fc5da0f5f39b5730 Integration tests for VPC-SC. (googleapis#48) by Qian Long <qlong@alum.mit.edu> - 8ce495510ff37cb27a1c840d212288cb5c87d327 Import of package:google-cloud-spanner to GitHub Private ... by Cloud Spanner Team <no-reply@google.com> - 8493b86 Re-generate library using google-cloud-clients/google-clo... by DPE bot <dpebot@google.com> - 5555939 Release v0.66.1-SNAPSHOT (googleapis#3784) by Jeff Ching <chingor@google.com> - ed6a3f5 Release google-cloud-java v0.66.0 (googleapis#3782) by Jeff Ching <chingor@google.com> - 1e539fa Cloud Spanner DML & PDML Release (googleapis#3781) by snehashah16 <snehashah@google.com> - d197058 add null check to JobInfo.fromPb(Job) and .toPb() (googleapis#3770) by sombra-mykola-bakay <30506381+sombra-mykola-bakay@users.noreply.github.com> - 211bfad Add synthtool scripts (googleapis#3765) by Jeff Ching <chingor@google.com> - 4813432 Add listDocuments() API (googleapis#3759) by Sebastian Schmidt <mrschmidt@google.com> - 0f3bc0e Don't raise empty Snapshots on resets (googleapis#3750) by Sebastian Schmidt <mrschmidt@google.com> - 63aff86 Remove CircleCI status, coveralls, and version eye badges... by Jeff Ching <chingor@google.com> - 6a8e5d6 Bump to next snapshot version (googleapis#3768) by Jeff Ching <chingor@google.com> - 1093938 Release 1.47.0/0.65.0 (googleapis#3767) by Jeff Ching <chingor@google.com> - f69710d Deprecate getCollections in favor of listCollections (googleapis#37... by Sebastian Schmidt <mrschmidt@google.com> - 7f74c5a [Storage] Add launch stage annotations around bucket lock... by Frank Natividad <frankyn@users.noreply.github.com> - 152653d Regenerate clients (googleapis#3766) by Jeff Ching <chingor@google.com> - 68f38e1 Revert "Revert "Revert "Cloud Spanner DML & PartitionedDM... by Jeff Ching <chingor@google.com> - f741fc0 Fix README typo (googleapis#3762) by ihommani <ismaelhommani@gmail.com> - 65ca2f3 Revert "Revert "Cloud Spanner DML & PartitionedDML suppor... by Jeff Ching <chingor@google.com> - 6ebbc19 fix incorrect link (googleapis#3748) by Elliotte Rusty Harold <elharo@users.noreply.github.com> - bcb28f2 [Storage] Bucket Lock (googleapis#3727) by Frank Natividad <frankyn@users.noreply.github.com> - c79f280 spanner: Expand test coverage for GrpcResultSet and runWi... by Nithin Sujir <118742+nithinsujir@users.noreply.github.com> - fdccb0b Bump to next snapshot version (googleapis#3744) by JesseLovelace <43148100+JesseLovelace@users.noreply.github.com> - 456e8fb Release 1.46.0/0.64.0 (googleapis#3742) by JesseLovelace <43148100+JesseLovelace@users.noreply.github.com> - 82fda75 spanner: Expand test coverage for SpannerClient (googleapis#3725) by Nithin Sujir <118742+nithinsujir@users.noreply.github.com> - 9187488 Fix parent/child spans relationship in Spanner. (googleapis#3690) by Bogdan Drutu <bdrutu@google.com> - 3c9bbf2 Revert "Cloud Spanner DML & PartitionedDML support (googleapis#3703... by JesseLovelace <43148100+JesseLovelace@users.noreply.github.com> - c163907 Cloud Spanner DML & PartitionedDML support (googleapis#3703) by snehashah16 <snehashah@google.com> - 626e4d1 Regenerate clients (googleapis#3738) by JesseLovelace <43148100+JesseLovelace@users.noreply.github.com> - ba388da Remove duplicated folders (googleapis#3732) by Jeff Ching <chingor@google.com> - d739242 spanner: Options: Fix null dereference, expand test cover... by Nithin Sujir <118742+nithinsujir@users.noreply.github.com> - 7499c31 Remove Circle CI and Travis configs (googleapis#3723) by Jeff Ching <chingor@google.com> - 7a72784 Remove dependency on vulnerable version of jackson, upgra... by JesseLovelace <43148100+JesseLovelace@users.noreply.github.com> - 2c284dd Add retries to downloadEmulator() to mitigate transient n... by JesseLovelace <43148100+JesseLovelace@users.noreply.github.com> - 6955469 bump proto version (googleapis#3718) by Hanzhen Yi <33737743+hzyi-google@users.noreply.github.com> - 9ab7770 Add ArrayUnion/ArrayRemove conformance tests (googleapis#3704) by Sebastian Schmidt <mrschmidt@google.com> - 36409f5 Bump version to 0.63.1-beta-SNAPSHOT (googleapis#3714) by JesseLovelace <43148100+JesseLovelace@users.noreply.github.com> - 94f19b7 Regenerate clients (googleapis#3713) by JesseLovelace <43148100+JesseLovelace@users.noreply.github.com> - 7a43f6f Release 1.45.0/0.63.0 (googleapis#3710) by JesseLovelace <43148100+JesseLovelace@users.noreply.github.com> - b533c9c Add region tags around an informative sample. (googleapis#3705) by Frank Natividad <frankyn@users.noreply.github.com> - e8cff3f Spanner: Block nested transactions (googleapis#3628) by Nithin Sujir <118742+nithinsujir@users.noreply.github.com> - 7869364 Migrate Pub/Sub client to ApiFutures. (googleapis#3700) by Anuraag Agrawal <anuraaga@gmail.com> - e310f6d ci: make javadoc not spam us (googleapis#3699) by Michael Darakananda <pongad@google.com> - fbf8715 Clean up snippets for BatchClient and Spanner (googleapis#3684) by Nithin Sujir <118742+nithinsujir@users.noreply.github.com> - 3fa1fa0 Bigtable: limit mutation sizes in the client to avoid ove... by Igor Bernstein <igorbernstein@google.com> - f977c8a bigquery: properly fail when setting TableId's project tw... by Marcello Steiner <MMMarcy@users.noreply.github.com> - c90b5ba Bigtable: add await replication (googleapis#3658) by Igor Bernstein <igorbernstein@google.com> - 43d53a8 Bigtable: wrap proto enums (googleapis#3659) by Igor Bernstein <igorbernstein@google.com> - 6f1a105 Bigtable: add enhanced stub for bigtable table admin clie... by Igor Bernstein <igorbernstein@google.com> - 68e1717 Kokoro additions (googleapis#3685) by Jeff Ching <chingor@google.com> - a175980 spanner: expand test coverage for getDatabaseClient() (googleapis#3... by Nithin Sujir <118742+nithinsujir@users.noreply.github.com> - ae614b3 pubsub: add Publisher.awaitTermination (googleapis#3688) by Michael Darakananda <pongad@google.com> - 739d519 Add downloadFile sample and reformat storage snippets (googleapis#3... by Frank Natividad <frankyn@users.noreply.github.com> - c901932 storage: include information on a bucket prefix (googleapis#3671) by Frank Natividad <frankyn@users.noreply.github.com> - 713cb88 link to google-cloud-logging from README (googleapis#3681) by Elliotte Rusty Harold <elharo@users.noreply.github.com> - 5ccfc3a spanner: Add snippets for ReadContext (googleapis#3662) by Nithin Sujir <118742+nithinsujir@users.noreply.github.com> - ccf68f3 Verify protoc version for batch-generation (googleapis#3676) by Andrea Lin <andrealin@google.com> - 1d7b95f Add Kokoro CI config (googleapis#3664) by Jeff Ching <chingor@google.com> - e855d48 Bump to next snapshot versions (googleapis#3679) by Garrett Jones <garrettjonesgoogle@users.noreply.github.com> - bbede73 Release 1.44.0/0.62.0 (googleapis#3677) by Garrett Jones <garrettjonesgoogle@users.noreply.github.com> - a420738 Weekly proto refresh (googleapis#3674) by Garrett Jones <garrettjonesgoogle@users.noreply.github.com> - 705acb7 update gax to 1.31/0.48 (googleapis#3675) by Hanzhen Yi <33737743+hzyi-google@users.noreply.github.com> - eadf7bc Adding redis-v1 and video-intelligence-v1p2beta1 (googleapis#3669) by Garrett Jones <garrettjonesgoogle@users.noreply.github.com> - f098439 Regenerate proto/grpc files with protoc 3.6.0 (googleapis#3672) by Andrea Lin <andrealin@google.com> - 4d1da48 Add redis-v1 and video-intelligence-v1p2beta1 to batch (#... by Garrett Jones <garrettjonesgoogle@users.noreply.github.com> - 80c9675 Fixes for ITComputeTest (googleapis#3667) by Andrea Lin <andrealin@google.com> - 509b079 Releasing.md instructions to uncomment nexus-staging-mave... by Andrea Lin <andrealin@google.com> - a2f7111 Add handwritten integration test for Compute GAPIC (googleapis#3660) by Andrea Lin <andrealin@google.com> - 0e58b7d bigtable: RowMutation should allow passing of a Mutation ... by ajaaym <34161822+ajaaym@users.noreply.github.com> - 37675e9 Bump to snapshot version for development (googleapis#3656) by Andrea Lin <andrealin@google.com> - e4b5266 Release 0.61.0/1.43.0 (googleapis#3653) by Andrea Lin <andrealin@google.com> - 20c329a Javadoc fixes for Bigtable client (googleapis#3652) by Andrea Lin <andrealin@google.com> - d235a82 tell JVM to use less memory when testing (googleapis#3650) by Michael Darakananda <pongad@google.com> - a9860b3 Bigtable: improve list tables spooler (googleapis#3639) by Igor Bernstein <igorbernstein@google.com> - 01bec2b Refresh all clients (googleapis#3647) by Andrea Lin <andrealin@google.com> - 0bef731 Bigtable: cosmetic cleanup of table admin (googleapis#3638) by Igor Bernstein <igorbernstein@google.com> - 3cefb4f Bigtable: table model improvements (googleapis#3640) by Igor Bernstein <igorbernstein@google.com> - 47bd674 Fix logging integration test failure on when running on G... by Hanzhen Yi <33737743+hzyi-google@users.noreply.github.com> - 124ed55 Revert "[Storage] Bucket lock (googleapis#3574)" (googleapis#3644) by Frank Natividad <frankyn@users.noreply.github.com> - 7f3ab97 Regenerate compute (googleapis#3642) by Andrea Lin <andrealin@google.com> - 9f1a96b [Storage] Bucket lock (googleapis#3574) by Frank Natividad <frankyn@users.noreply.github.com> - 5e3b897 pubsub: clean up after extension gives up (googleapis#3633) by Chris Sainty <csainty@users.noreply.github.com> - f4bc56d make DatastoreBatchWriter public (googleapis#3387) by Daniel Norberg <daniel.norberg@gmail.com> - c81ff3e Bigtable: add resource level IAM (googleapis#3624) by Igor Bernstein <igorbernstein@google.com> - c715fc6 Update signUrl documentation (googleapis#3546) by Jeff Ching <chingor@google.com> - d7a135a google-cloud-nio: retry on 502 errors, and increase max d... by droazen <droazen@broadinstitute.org> - 1f12a83 spanner: Add snippets for Spanner, BatchClient and BatchR... by Nithin Sujir <118742+nithinsujir@users.noreply.github.com> - 997d2a3 Bigtable: add CRUD for AppProfiles (googleapis#3619) by Igor Bernstein <igorbernstein@google.com> - e3eedeb Bigtable: add CRUD for clusters (googleapis#3612) by Igor Bernstein <igorbernstein@google.com> - 8bcc89b Bigtable: clean up consistency token (googleapis#3570) by Igor Bernstein <igorbernstein@google.com> - 427b155 Bigtable: add CRUD for instances (googleapis#3569) by Igor Bernstein <igorbernstein@google.com> - b9b4aff fix logging unit tests (googleapis#3630) by Hanzhen Yi <33737743+hzyi-google@users.noreply.github.com> - ff2ed5e bump to snapshot version (googleapis#3625) by Hanzhen Yi <33737743+hzyi-google@users.noreply.github.com> - 4cd518d Release 0.60.0 (googleapis#3623) by Hanzhen Yi <33737743+hzyi-google@users.noreply.github.com> - 3d1e971 batch generation before release (googleapis#3622) by Hanzhen Yi <33737743+hzyi-google@users.noreply.github.com> - 08a5e46 [Storage] Make StorageIT easier to setup with new project... by Frank Natividad <frankyn@users.noreply.github.com> - 4eca7a4 upgrade auth version (googleapis#3606) by Elliotte Rusty Harold <elharo@users.noreply.github.com> - 8f44a9a removing word "natural" from product name (googleapis#3610) by Vincent <vtsao@users.noreply.github.com> - 4c3e3d1 Fix documentation for setParallelPullCount (googleapis#3542) by Jonas Wiklund <jwiklund@gmail.com> - 5fc0dbd bump version for development (googleapis#3599) by Michael Darakananda <pongad@google.com> (And 1760 more changes) GitOrigin-RevId: b2056bd147466fec8a817885fc5da0f5f39b5730

snehashah16 added 2 commits September 18, 2018 14:51

Support for Cloud Spanner DML & PDML

c62374b

Fix for DML/PDML - manual error from sync

4e2a950

snehashah16 requested a review from pongad as a code owner September 19, 2018 17:53

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Sep 19, 2018

snehashah16 requested a review from vam-google September 19, 2018 17:54

snehashah16 added 2 commits September 19, 2018 13:32

Codacybot & Stylecheck fixes.

f1bf1be

Fix merge conflict

44558bb

danoscarmike changed the title ~~WIP: Cloud Spanner DML & PDML launch~~ WIP: Cloud Spanner feature launch Sep 20, 2018

danoscarmike added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Sep 20, 2018

vam-google reviewed Sep 21, 2018

View reviewed changes

snehashah16 added 2 commits September 24, 2018 15:35

PR pull/3703 fixes

8dfd720

Merge branch 'master' into snehashah-dml

50221d0

snehashah16 commented Sep 24, 2018

View reviewed changes

vam-google approved these changes Sep 26, 2018

View reviewed changes

snehashah16 removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Sep 27, 2018

snehashah16 changed the title ~~WIP: Cloud Spanner feature launch~~ Cloud Spanner DML & PartitionedDML support Sep 27, 2018

snehashah16 merged commit c163907 into master Sep 27, 2018

JesseLovelace added a commit to JesseLovelace/google-cloud-java that referenced this pull request Sep 27, 2018

Revert "Cloud Spanner DML & PartitionedDML support (googleapis#3703)"

ba8f7c6

This reverts commit c163907.

JesseLovelace added a commit that referenced this pull request Sep 27, 2018

Revert "Cloud Spanner DML & PartitionedDML support (#3703)" (#3741)

3c9bbf2

This reverts commit c163907.

chingor13 added a commit to chingor13/google-cloud-java that referenced this pull request Oct 1, 2018

Revert "Revert "Cloud Spanner DML & PartitionedDML support (googleapi…

02cc5da

…s#3703)" (googleapis#3741)" This reverts commit 3c9bbf2.

chingor13 added a commit that referenced this pull request Oct 2, 2018

Revert "Revert "Cloud Spanner DML & PartitionedDML support (#3703)" (#…

65ca2f3

…3741)" (#3755) This reverts commit 3c9bbf2.

chingor13 added a commit to chingor13/google-cloud-java that referenced this pull request Oct 3, 2018

Revert "Revert "Revert "Cloud Spanner DML & PartitionedDML support (g…

dbf7f64

…oogleapis#3703)" (googleapis#3741)" (googleapis#3755)" This reverts commit 65ca2f3.

chingor13 added a commit that referenced this pull request Oct 3, 2018

Revert "Revert "Revert "Cloud Spanner DML & PartitionedDML support (#…

68f38e1

…3703)" (#3741)" (#3755)" (#3761) This reverts commit 65ca2f3.

snehashah16 added a commit that referenced this pull request Oct 5, 2018

Launch Cloud Spanner DML & PartitionedDML support

1c68385

Revert "Revert "Revert "Revert "Cloud Spanner DML & PartitionedDML support (#3703)" (#3741)" (#3755)" (#3761)" This reverts commit 68f38e1.

chingor13 deleted the snehashah-dml branch October 16, 2018 20:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cloud Spanner DML & PartitionedDML support #3703

Cloud Spanner DML & PartitionedDML support #3703

snehashah16 commented Sep 19, 2018

vam-google commented Sep 19, 2018

snehashah16 commented Sep 19, 2018 •

edited

Loading

snehashah16 commented Sep 21, 2018

vam-google left a comment

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

snehashah16 left a comment

vam-google left a comment •

edited

Loading

This comment was marked as spam.

This comment was marked as spam.

Cloud Spanner DML & PartitionedDML support #3703

Cloud Spanner DML & PartitionedDML support #3703

Conversation

snehashah16 commented Sep 19, 2018

vam-google commented Sep 19, 2018

snehashah16 commented Sep 19, 2018 • edited Loading

snehashah16 commented Sep 21, 2018

vam-google left a comment

Choose a reason for hiding this comment

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

snehashah16 left a comment

Choose a reason for hiding this comment

vam-google left a comment • edited Loading

Choose a reason for hiding this comment

This comment was marked as spam.

This comment was marked as spam.

snehashah16 commented Sep 19, 2018 •

edited

Loading

vam-google left a comment •

edited

Loading