Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pass on azure sdk keyword arguments #376

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

ptrba
Copy link

@ptrba ptrba commented Dec 1, 2022

Motivation

The current implementation wraps azure-storage-blob methods without giving access to advanced keyword parameters. In rare but some cases (#375) it is necessary to tune those parameters. But, passing on all keyword arguments to azure SDK methods has potentially unintended effects and possible conflicts as discussed in (draft #364).

Suggestion here

  • There is a tool (tools/azure_sdk_kwargs.py), which checks all azure SDK methods for potential conflicts and generates a docstring for the AzureBlobFileSystem constructor. This is only a semi-automated process, but good enough. It only needs to be rerun for major changes in the azure SDK, which means touching the code anyway.
  • Known conflicting arguments are explicitly excluded from being passed on to the azure SDK methods.

This solution is sufficiently safe considering the fact that this is an advanced feature. At the same time, it is flexible enough to deal with changes in the azure SDK contracts. The docstring points to a specific version of the azure SDK. If the user decides to work with a different SDK or wants to use hidden undocumented arguments, he can use any keyword argument (except for the explicitly excluded ones) if he knows what he's doing.

This is a cleanup of the ideas discussed in draft #364.

Peter Barmettler added 2 commits December 1, 2022 16:43
commit ab4ca54
Author: Peter Barmettler <peter.barmettler@raccoonworks.ch>
Date:   Thu Dec 1 16:31:40 2022 +0100

    auto use storage

commit d5bdd8c
Author: Peter Barmettler <peter.barmettler@raccoonworks.ch>
Date:   Thu Dec 1 16:26:32 2022 +0100

    azure sdk docstring | pass on parameters | basic test

commit 8fdb60b
Author: Peter Barmettler <peter.barmettler@raccoonworks.ch>
Date:   Thu Dec 1 15:19:03 2022 +0100

    Revert "client and upload kwargs"

    This reverts commit cca3c5e.

commit 8a71e55
Author: Peter Barmettler <peter.barmettler@raccoonworks.ch>
Date:   Thu Dec 1 15:17:58 2022 +0100

    add azure sdk docstring

commit cca3c5e
Author: Peter Barmettler <peter.barmettler@raccoonworks.ch>
Date:   Thu Oct 27 12:49:12 2022 +0200

    client and upload kwargs
@ptrba ptrba changed the title Pass on azure sdk keyworkd arguments Pass on azure sdk keyword arguments Dec 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant