feat: Transaction propagation using ndb.TransactionOptions

[JohnGale87]

Hi Cloud NDB team,

I did review your CONTRIBUTING.rst but if I've done anything wrong then please let me know and I will of course endevour to correct it. I've never done this before so wasn't sure of the process - do I just submit a pull request straight away or am I supposed to open an issue first...?

Anyway, our application uses the existing google.appengine.ext.ndb library and in doing so makes relatively extensive use of the propagation options from ndb.TransactionOptions. So knowing that you marked them all as NoLongerImplementedError has been a bit of a blocker for us migrating off of python 2.

A few months ago I followed closely on #366 and that also lead me to read #4.

Having now read those issues and from what I understand of the discussion(s) they contained combined with my observations in both the old and new implementations, the main reason I can discern for the new library not including the original propagation options seems mostly to do with the ndb.TransactionOptions.NESTED option. Which the documentation says isn't actually supported by the old AppEngine python ndb library anyway...

The NESTED policy is not supported. If you use this policy, your code will throw a BadRequestError exception.
https://cloud.google.com/appengine/docs/standard/python/ndb/functions#context_options

In #381 you added the join parameter and for @ndb.transactional set the default to be True which seems to me to be the equivalent of ndb.TransactionOptions.ALLOWED being the default propagation option in the old library (please correct me if I'm wrong).

Based on those assumptions I have been looking at the feasibility of adding equivalents of ndb.TransactionOptions.MANDATORY and ndb.TransactionOptions.INDEPENDENT. I think I have a solution for doing so (presented in this pull request) but wanted to have my thoughts validated and my code reviewed and (likely) refined.

My implementation currently maintains compatibility with the join argument (I think) so that any users who have already started to use that will not be impacted by these changes. Although further changes would make it possible for the propagation argument to supersede and replace the join argument if wanted.

The implementation I am most unsure about is the ndb.TransactionOptions.INDEPENDENT one. I used a similar transaction pausing technique as was used in non_transactional but in the newly created context I set some additional arguments to None in an effort to ensure that the new transaction shares nothing with the previous one. Admittedly I'm not 100% sure if I should have done that or not, or if I got all the right arguments, but I'm hoping your review will reveal the answer to that.

new_ctx = ctx.new(transaction=None,
                  batches=None,
                  commit_batches=None,
                  cache=None)

[google-cla]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

---

What to do if you already signed the CLA

Individual signers

• It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

• Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
• The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

[JohnGale87]

@googlebot I signed it!

[andrewsg]

Apologies for the long delay. I'm reaching out to the Datastore backend team for details on this feature and will be back with some more info as soon as that is resolved.

[JohnGale87]

Is there any update on this that you can share with me? Please do let me know if I can be of assistance in any way.

[andrewsg]

@JohnGale87 Thanks again for your patience. We were initially concerned that this behavior should be tying into server-side functionality we were missing, but we've looked into it and it looks like it should be here in the client after all.

Thanks for including tests as well! We'll review this PR for inclusion.

[cguardia]

Thanks! Please take a look at the comments. This is almost ready to merge.

[cguardia]

@JohnGale87 I ran the tests again and noticed that in addition to Python 2.7 compatibility, this PR needs to get to 100% unit test coverage and pass the lint checks (run nox -e blacken, then nox -e lint). Thanks for all your effort.

[cguardia]

@JohnGale87 Thanks for your recent changes! Getting closer, lint and python 2.7 compatibility are passing now. The test coverage is still not 100% however. Could you please work in a few more tests (run nox -e cover for info)? Also, a docstring for the class would be great. Thanks!

[JohnGale87]

@cguardia Yes sorry, I'm in the habit of pushing every time I commit, didn't realise it would be spamming you. I am still working on it. I'll try to stop pushing to git until I'm finished!

[chrisrossi]

See notes.

[chrisrossi]

Excellent work, thank you! Just a couple of editing notes on the docs.

[chrisrossi]

Thank you!

[chrisrossi]

I believe @andrewsg or @crwilcox will have to merge.