feat: new transaction feature #1239

danieljbruce · 2024-03-04T19:17:32Z

Summary

This PR ensures that transaction read calls will use newTransaction options to begin transactions instead of making a separate beginTransaction grpc call.

Source code changes

A #blockWithMutex function is added and it is used for read calls instead of a #withBeginTransaction function. This makes it so that all read calls on Transaction will block on the mutex and then call the same method in their super class. Before this change, the read calls on Transaction would block on the mutex, make a beginTransaction grpc call and then call the same method in their super class. So this change to using #blockWithMutex omits the extra beginTransaction call and instead expects the function it is calling to supply new transaction options.

A function getTransactionRequest is added which pulls out the code used for building transaction options for a beginTransaction request. That is because this code is also used to build the new transaction options.

TransactionState is moved up to the super class DatastoreRequest and a new state is added called NOT_TRANSACTION for DataRequests that are not transactions.

parseTransactionResponse is moved up to the super class DatastoreRequest and is now used by runQuery, runAggregationQuery and get to save the transaction id.

newTransaction options are now built in getRequestOptions which is used by runQuery, runAggregationQuery and get to save the transaction id.

state is moved up to the DatastoreRequest super class because it now has to be used to decide whether or not to build newTransaction options.

If a transaction is expired then an error will be thrown when the transaction is used.

ReadOptions are the options shared by Datastore read requests.
new_transaction is a consistency type in ReadOptions. When new_transaction is used in a read request, a new transaction is begun for the request.

System tests added

A system test is added to ensure using new transaction has better latency. System tests are added for runQuery, runAggregationQuery and get for readOnly transactions to ensure readOnly transactions work correctly whether transaction.run is called or not.

Unit tests added

Unit tests are added for various orders of transaction operations that ensure correct behaviour for when transaction.run is used and when transaction.run is omitted. lookup, lookup, put, commit is one order that is tested. The others are runQuery, lookup, put, commit, runAggregationQuery, lookup, put, commit, put, put, lookup, commit and put, commit. A unit test is also corrected.

Drawbacks

If the user uses transaction.runQueryStream or they use transaction.createReadStream then the call will not block on a mutex. These methods are not async methods so they cannot wait for a #blockWithMutex call to complete. If users are currently using runQueryStream or createReadStream concurrently then these multiple read calls may be beginning multiple transactions at the same time using one transaction object which is not ideal.

Transaction state should move up to the super class

Add a method for building transaction options. Also add a method for detecting if a request is a transaction. Build new transaction options

Corrected getTransactionRequest. Build the request options properly.

When this system test isn’t set to be a readOnly transaction then the error occurs which says too much contention.

Change parseRunSuccess so that it can be used more universally.

parseRunSuccess should move up to the super class because it now needs to be used there.

This change saves the transaction id returned from the server for read calls

Read only tests are needed

Make sure that code using the new transaction option has better performance than code that doesn’t have it.

Use the MockedTransactionWrapper to test requests being passed into the Gapic layer.

Mock out begin transaction. Test for newTransaction consistency type. Mock out commit.

A transaction.run test is needed for lookup, lookup, put, commit.

Add a test for this sequence of operations and ensure it works properly.

Four operations that get all the results for running an aggregation query. Adding another test for these four operations.

Last test suite regarding new transaction unit tests.

Add a bunch of tests for the commit case. Check the commit gapic input.

Begin transaction should be called at least once. Add code here to increment the counter.

Remove commented code too.

Check for expired on most functions and write tests for the expired check.

…anieljbruce/nodejs-datastore into new-transaction-feature-branch

danieljbruce · 2024-06-05T18:05:58Z

test/gapic-mocks/runQuery.ts

@@ -68,35 +53,13 @@ describe('Run Query', () => {
    }
  }

-  it('should pass read time into runQuery for transactions', async () => {


Note that this test is no longer valid because transaction and read time are no longer compatible together as is the case in Python.

cindy-peng · 2024-06-06T03:09:28Z

src/request.ts

+      throw Error(transactionExpiredError);
+    }
+  }
+
  /**
   * Retrieve the entities as a readable object stream.
   *


Nit: update L381 with new added errors?

Sure. Done.

cindy-peng · 2024-06-06T06:02:25Z

src/transaction.ts

+      callback(new Error(transactionExpiredError));
+      return;
+    }
+    if (this.state === TransactionState.NOT_STARTED) {


Just wondering what are the allowed states here? NOT_TRANSACTION and IN_PROGRESS?

NOT_TRANSACTION, NOT_STARTED, IN_PROGRESS, EXPIRED.

cindy-peng · 2024-06-06T06:03:09Z

src/request.ts

+    if (this.state === TransactionState.EXPIRED) {
+      callback(new Error(transactionExpiredError));
+      return;
+    }


Do we need to check readTimeAndConsistencyError here?

No. That check is done further downstream in the call to runQueryStream. But I did spot some other cases where errors weren't bubbling up and fixed those bugs.

Fix the concurrency tests and move the error reporting outside of the stream, add two tests for the streams to make sure errors get reported.

This makes it so that it is reported in the streams.

daniel-sanche

LGTM

daniel-sanche · 2024-06-18T18:49:43Z

src/request.ts

+export function getTransactionRequest(
+  transaction: Transaction,
+  options: RunOptions
+): TransactionRequestOptions {


please add some comments

daniel-sanche · 2024-06-18T18:58:10Z

src/request.ts

+    if (id) {
+      reqOpts.readWrite = {previousTransaction: id};
+    }
+  }


is there a way to trim down this logic a bit? Seems a bit redundant

I generated unit tests, corrected them and then reduced this function down to one ternary operation. Previously this was just copy pasted code from before. Search describe('getTransactionRequest', () => { to see how this function should behave.

I am going to rewrite the getTransactionRequest function so it is a good idea not to change how it works.

Fix the compiler error Change code so that when readOnly is specified then readwrite options are ignored. Fix the unit test to adopt this change.

Add comments, simplify logic in the function.

…anieljbruce/nodejs-datastore into new-transaction-feature-branch

danieljbruce added 2 commits March 4, 2024 10:59

Move the state up, make it protected

f9bb82e

Transaction state should move up to the super class

Build new transaction options method

e6c1265

Add a method for building transaction options. Also add a method for detecting if a request is a transaction. Build new transaction options

product-auto-label bot added size: m Pull request size is medium. api: datastore Issues related to the googleapis/nodejs-datastore API. labels Mar 4, 2024

danieljbruce added 9 commits March 4, 2024 14:53

Correct the fn that builds transactions

974eee2

Corrected getTransactionRequest. Build the request options properly.

Set transaction to readOnly in system test.

9cf180a

When this system test isn’t set to be a readOnly transaction then the error occurs which says too much contention.

Change #parseRunSuccess

a58d4ec

Change parseRunSuccess so that it can be used more universally.

Move parseRunSuccess up to protected

bf110eb

parseRunSuccess should move up to the super class because it now needs to be used there.

Saves the transaction id

f1cb78d

This change saves the transaction id returned from the server for read calls

Add tests for testing read only transactions

8a420c3

Read only tests are needed

Add tests that measure read time

6ca70ac

Make sure that code using the new transaction option has better performance than code that doesn’t have it.

ran linter

46121e0

Add a test for testing requests

8b5a311

Use the MockedTransactionWrapper to test requests being passed into the Gapic layer.

product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Mar 6, 2024

danieljbruce added 7 commits March 6, 2024 13:52

Final changes to make test work

5a91c4e

Mock out begin transaction. Test for newTransaction consistency type. Mock out commit.

Add the transaction.run test

38d76ff

A transaction.run test is needed for lookup, lookup, put, commit.

run linter

6299bf3

runQuery, lookup, put, commit

e36b158

Add a test for this sequence of operations and ensure it works properly.

runAggregationQuery, lookup, put, commit

28a89ed

Four operations that get all the results for running an aggregation query. Adding another test for these four operations.

put, put, lookup, commit

fa4f200

Last test suite regarding new transaction unit tests.

Add tests for the commits

6c976e1

Add a bunch of tests for the commit case. Check the commit gapic input.

product-auto-label bot added size: xl Pull request size is extra large. and removed size: l Pull request size is large. labels Mar 6, 2024

danieljbruce added 3 commits March 7, 2024 09:58

Add testing to ensure begin tx is called

920f282

Begin transaction should be called at least once. Add code here to increment the counter.

Document #blockWithMutex

7f9c2c9

Remove commented code too.

Document transaction state

f762bb1

danieljbruce changed the title ~~New transaction feature branch~~ feat: new transaction feature branch Mar 7, 2024

danieljbruce added 2 commits March 7, 2024 15:21

feat: new transaction feature branch

4375378

Add a check for expired

599bec1

Check for expired on most functions and write tests for the expired check.

danieljbruce and others added 6 commits June 5, 2024 11:09

Run. linter

8091062

Merge branch 'main' into new-transaction-feature-branch

4a74f4d

read time and consistency error

d52c773

Merge branch 'new-transaction-feature-branch' of https://github.com/d…

66b0286

…anieljbruce/nodejs-datastore into new-transaction-feature-branch

Remove unnecessary change

2b93852

Eliminate the call to withBeginTransaction

22e7b9d

danieljbruce commented Jun 5, 2024

View reviewed changes

cindy-peng reviewed Jun 6, 2024

View reviewed changes

danieljbruce added 4 commits June 10, 2024 13:43

Throw error reported in documentation

a54b6c6

Work on streaming errors

418c43b

Fix the concurrency tests and move the error reporting outside of the stream, add two tests for the streams to make sure errors get reported.

Move the error throwing up the stack

2edd410

This makes it so that it is reported in the streams.

Fix linting errors

5c1dee5

danieljbruce changed the title ~~feat: new transaction feature branch~~ feat: new transaction feature Jun 18, 2024

daniel-sanche approved these changes Jun 18, 2024

View reviewed changes

danieljbruce and others added 7 commits June 20, 2024 11:36

Generate unit tests for getTransactionRequest

7bf7a90

I am going to rewrite the getTransactionRequest function so it is a good idea not to change how it works.

Generate unit tests

2022006

Fix the compiler error Change code so that when readOnly is specified then readwrite options are ignored. Fix the unit test to adopt this change.

Merge branch 'main' into new-transaction-feature-branch

f94c37e

Make function one ternary operation

03b7cd3

Add comments, simplify logic in the function.

Merge branch 'new-transaction-feature-branch' of https://github.com/d…

b16afbf

…anieljbruce/nodejs-datastore into new-transaction-feature-branch

Add two comments

9b05013

Update the comment

f10d1ef

sofisl approved these changes Jun 21, 2024

View reviewed changes

Run the linter

7e453e1

danieljbruce added the owlbot:run Add this label to trigger the Owlbot post processor. label Jun 24, 2024

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Jun 24, 2024

danieljbruce merged commit cdd2ee9 into googleapis:main Jun 24, 2024
15 checks passed

release-please bot mentioned this pull request Jun 24, 2024

chore(main): release 9.1.0 #1252

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: new transaction feature #1239

feat: new transaction feature #1239

danieljbruce commented Mar 4, 2024 •

edited

Loading

danieljbruce Jun 5, 2024

cindy-peng Jun 6, 2024

danieljbruce Jun 10, 2024

cindy-peng Jun 6, 2024

danieljbruce Jun 10, 2024

cindy-peng Jun 6, 2024

danieljbruce Jun 11, 2024

daniel-sanche left a comment

daniel-sanche Jun 18, 2024

danieljbruce Jun 20, 2024

daniel-sanche Jun 18, 2024

danieljbruce Jun 20, 2024

feat: new transaction feature #1239

feat: new transaction feature #1239

Conversation

danieljbruce commented Mar 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daniel-sanche left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danieljbruce commented Mar 4, 2024 •

edited

Loading