-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Batch dataset.put and dataset.delete. #421
Comments
Given a list of entities in hand, what is the advantage of a "convenience" method on the dataset to save or delete them, against just iterating the list and calling I guess there might be some utility to a |
I think |
The Zen of Python says:
What use is
If |
It's mostly for performances reason. Doing:
Means one RPC for each |
Isn't that what you get from using a transaction? All the adds / saves / deletes get sent as a single RPC. |
That's meant for a transactional context, where if one fails, they all fail. |
I think it makes sense to have this be a mode/flag/subclass of
|
IMO we need to focus on feature parity with the API before we can add nice sugar like that. As @silvolu says, we need methods for batch puts and deletes. |
What is the expectation for error propagation after a non-transactional batch update / delete, where some of the operations failed? |
@tseaver There is none. If guarantees need to be made about mutation success transactions must be used. @dhermes It's not necessarily sugar. If instead of immediately calling This implementation also works quite nicely for transactions. I have a class hierarchy I made for the |
@elibixby are you saying there is no way to detect the failures in batch mode? Who would use an API like that, and why would we go out of our way to support them? |
@tseaver this is fundamentally a property of the datastore, not the API. You could for instance make a bunch of batch writes, and then do other stuff. Then when you need to use the data, in a transaction retrieve all of the writes you made, redo the writes that failed, and operate on your data as normal. This will be faster than simply making all the writes in the transaction, as fewer writes will have to propagate through the datastore, (since transactions demand snapshot isolation and strong consistency) EDIT: to expand on this, transactions are for making mutations that are dependent upon a certain state in the datastore (e.g. updating data based on it's current value) OTOH you may want to make a huge number of mutations that don't depend on the state of the datastore. If you want to add 10,000 entities to the datastore (during a short time, and not dependent on the current content of the datastore), it would be EXTREMELY bad to slam the datastore with 10,000 api calls. This is a frequent use case during datastore initialization or clean up (in testing for example) |
@dhermes we have examples of context-manager-based batching in Storage already:
|
@tseaver with the major refactorings, is this still under the radar? |
It's right on the horizon (for one of us at least). |
This is a pre-cursor to designated put(), delete() and get() methods in this module (inspired by googleapis#421).
This is a pre-cursor to designated put(), delete() and get() methods in this module (inspired by googleapis#421).
This is a pre-cursor to designated put(), delete() and get() methods in this module (inspired by googleapis#421).
This is a pre-cursor to designated put(), delete() and get() methods in this module (inspired by googleapis#421).
This is a pre-cursor to designated put(), delete() and get() methods in this module (inspired by googleapis#421).
* chore(deps): update all dependencies * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * revert Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <partheniou@google.com>
* chore(deps): update all dependencies * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * revert Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <partheniou@google.com>
* chore(deps): update all dependencies * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * revert Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <partheniou@google.com>
Source-Link: googleapis/synthtool@6ed3a83 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:3abfa0f1886adaf0b83f07cb117b24a639ea1cb9cffe56d43280b977033563eb Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Source-Link: googleapis/synthtool@fdba3ed Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:1f0dbd02745fb7cf255563dab5968345989308544e52b7f460deadd5e78e63b0
* feat: Add support for python 3.11 chore: Update gapic-generator-python to v1.8.0 PiperOrigin-RevId: 500768693 Source-Link: googleapis/googleapis@190b612 Source-Link: googleapis/googleapis-gen@7bf29a4 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiN2JmMjlhNDE0YjllY2FjMzE3MGYwYjY1YmRjMmE5NTcwNWMwZWYxYSJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
…p/templates/python_library/.kokoro (#421) Source-Link: https://github.com/googleapis/synthtool/commit/bb171351c3946d3c3c32e60f5f18cee8c464ec51 Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-python:latest@sha256:f62c53736eccb0c4934a3ea9316e0d57696bb49c1a7c86c726e9bb8a2f87dadf
* docs: add detect event with event input snippet * Update detect_intent_event_test.py
* fix: Add async context manager return types chore: Mock return_value should not populate oneof message fields chore: Support snippet generation for services that only support REST transport chore: Update gapic-generator-python to v1.11.0 PiperOrigin-RevId: 545430278 Source-Link: googleapis/googleapis@601b532 Source-Link: googleapis/googleapis-gen@b3f18d0 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYjNmMThkMGY2NTYwYTg1NTAyMmZkMDU4ODY1ZTc2MjA0NzlkN2FmOSJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
- [ ] Regenerate this pull request now. PiperOrigin-RevId: 402401837 Source-Link: googleapis/googleapis@16ff813 Source-Link: googleapis/googleapis-gen@c9e6ac2 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYzllNmFjMjVkNWUzMDkxNzBmMTgwNWJmOGVhMzgyYThiMjNkZWU2NSJ9
🤖 I have created a release \*beep\* \*boop\* --- ## [2.10.0](https://www.github.com/googleapis/python-dialogflow/compare/v2.9.1...v2.10.0) (2021-11-12) ### Features * add context manager support in client ([#416](https://www.github.com/googleapis/python-dialogflow/issues/416)) ([317187c](https://www.github.com/googleapis/python-dialogflow/commit/317187cbaacc6889d6fff5d7ea483fe1bc2cd9ee)) * add document metadata filter in article suggestion ([#437](https://www.github.com/googleapis/python-dialogflow/issues/437)) ([56a6e11](https://www.github.com/googleapis/python-dialogflow/commit/56a6e11622f73c6d302a5f43142ceb289b334fd1)) * add smart reply model in human agent assistant ([56a6e11](https://www.github.com/googleapis/python-dialogflow/commit/56a6e11622f73c6d302a5f43142ceb289b334fd1)) * add support for python 3.10 ([#422](https://www.github.com/googleapis/python-dialogflow/issues/422)) ([652e2e8](https://www.github.com/googleapis/python-dialogflow/commit/652e2e8d860f369b62e7866d6cf220204740ade8)) * **v2:** added support to configure security settings, language code and time zone on conversation profile ([#431](https://www.github.com/googleapis/python-dialogflow/issues/431)) ([6296673](https://www.github.com/googleapis/python-dialogflow/commit/629667367d7098cfb62bae1b6e48cc11a72b9fbc)) ### Bug Fixes * **deps:** drop packaging dependency ([fd06e9f](https://www.github.com/googleapis/python-dialogflow/commit/fd06e9fe8626ac3d86175518c52ff14efebc0f7b)) * **deps:** require google-api-core >= 1.28.0 ([fd06e9f](https://www.github.com/googleapis/python-dialogflow/commit/fd06e9fe8626ac3d86175518c52ff14efebc0f7b)) ### Documentation * clarified meaning of the legacy editions ([#426](https://www.github.com/googleapis/python-dialogflow/issues/426)) ([d7a7544](https://www.github.com/googleapis/python-dialogflow/commit/d7a7544ce69cb357d7cad13e9a44afe26c6d3cf5)) * clarified semantic of the streaming APIs ([d7a7544](https://www.github.com/googleapis/python-dialogflow/commit/d7a7544ce69cb357d7cad13e9a44afe26c6d3cf5)) * list oneofs in docstring ([fd06e9f](https://www.github.com/googleapis/python-dialogflow/commit/fd06e9fe8626ac3d86175518c52ff14efebc0f7b)) * **samples:** Added comments ([#425](https://www.github.com/googleapis/python-dialogflow/issues/425)) ([f5d40dc](https://www.github.com/googleapis/python-dialogflow/commit/f5d40dc9b4bb57b8830dcd6541a2a1189a6c9780)) * **v2beta1:** clarified meaning of the legacy editions ([fd06e9f](https://www.github.com/googleapis/python-dialogflow/commit/fd06e9fe8626ac3d86175518c52ff14efebc0f7b)) * **v2beta1:** clarified semantic of the streaming APIs ([fd06e9f](https://www.github.com/googleapis/python-dialogflow/commit/fd06e9fe8626ac3d86175518c52ff14efebc0f7b)) * **v2beta1:** recommend AnalyzeContent for future users ([#420](https://www.github.com/googleapis/python-dialogflow/issues/420)) ([1afdab3](https://www.github.com/googleapis/python-dialogflow/commit/1afdab3b50c98cc082b150ff408d0f07f11f9cf3)) * **v2:** recommend AnalyzeContent for future users ([#421](https://www.github.com/googleapis/python-dialogflow/issues/421)) ([c6940a9](https://www.github.com/googleapis/python-dialogflow/commit/c6940a9f974af95037616bd1affb34d8db4405c9)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
* feat: Add support for python 3.11 chore: Update gapic-generator-python to v1.8.0 PiperOrigin-RevId: 500768693 Source-Link: googleapis/googleapis@190b612 Source-Link: googleapis/googleapis-gen@7bf29a4 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiN2JmMjlhNDE0YjllY2FjMzE3MGYwYjY1YmRjMmE5NTcwNWMwZWYxYSJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
* feat: Update Compute Engine API to revision 20230701 (#821) Source-Link: googleapis/googleapis@761c3cb Source-Link: googleapis/googleapis-gen@cac56a0 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiY2FjNTZhMGVkZDExNzllOTI5YzMxZWJlYWFlODQ4N2YzODhkYzEwOSJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
* chore: Update gapic-generator-python to v1.11.4 PiperOrigin-RevId: 547897126 Source-Link: googleapis/googleapis@c09c75e Source-Link: googleapis/googleapis-gen@45e0ec4 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNDVlMGVjNDM0MzUxN2NkMGFhNjZiNWNhNjQyMzJhMTgwMmMyZjk0NSJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: add git_file_source and git_repo_source to build_trigger PiperOrigin-RevId: 550012872 Source-Link: googleapis/googleapis@f90d153 Source-Link: googleapis/googleapis-gen@7682e23 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNzY4MmUyMzFiNjA1OGFhNDM5YjRiNGY2ZGYyMzYyNDBiZjg4YjNlMiJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: Add automap_substitutions flag to use substitutions as envs in Cloud Build PiperOrigin-RevId: 551218480 Source-Link: googleapis/googleapis@f823915 Source-Link: googleapis/googleapis-gen@5979eec Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNTk3OWVlY2Q4ZDJiMGZjMTU4ZmU3MTUyYTRkYTQzY2VkMjkyOTc4MCJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * feat: add update_mask to UpdateBuildTriggerRequest proto PiperOrigin-RevId: 552479161 Source-Link: googleapis/googleapis@f8415bd Source-Link: googleapis/googleapis-gen@4ac7667 Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiNGFjNzY2N2M3ZGVhMzg3MzhkOWUyMTc2MzA1YTgwNGRlMjI1OGJlMSJ9 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * fix docs build. Issue filed upstream here googleapis/gapic-generator-python#1724 --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <partheniou@google.com>
* chore(deps): update all dependencies * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * revert Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <partheniou@google.com>
Like dataset.get, there should be a dataset.delete and dataset.put that accepts a list of keys / entities.
The text was updated successfully, but these errors were encountered: