-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Generic Secure Block #14899
Comments
Hey @Bryan-Meier and thank you for the request! It makes sense to me. For my own understanding I'd like to offer up two alternatives and see if either of them appeal to you - either way I'm happy to support a Option 1: Custom ClassYou could create a new Block type yourself, e.g., from prefect.blocks.core import Block
from pydantic import SecretStr
class MySecretDict(Block):
username: SecretStr
password: SecretStr
url: SecretStr
my_block = MySecretDict(username="foo", password="bar", url="blah")
my_block.save("example") I'm guessing this isn't ideal because you then need to have the importable class anywhere in which you want to use this block document. Option 2: ignoring block methods on loadThis option is not currently implemented so is more of a thought experiment to see if it would extend the pattern here and resolve other forms of frustration. The main idea is to reference blocks purely as typed-documents and ignore the business logic methods that tend to be on many of our blocks. First you would still register a new block type / document with the API as above. The big difference would be that loading this block would not require any imports, so you wouldn't need to redefine the class anywhere: from prefect.blocks.core import Block, inject_block_document
# this block would load the fields as a _pydantic model_ but not attempt to import the
# class itself
my_block = Block.load("mysecretdict/example", document_only=True)
# this could be extended to a decorator that injects fields into custom classes
@inject_block_document("mysecretdict/example")
class MyRandomClass:
pass
# would load the fields from the API and inject them as attrs onto this class instance
my_class = MyRandomClass() At your convenience I'm curious for your thoughts (or anyone else lurking!) on either of those approaches. |
Hi @cicdw, The comment you made at the end of option 1 is exactly what we are running into. We want to create the block type and then use it anywhere we desire without having to copy the definition of the block into every Prefect job that we plan to use it in. There are definitely ways to do this and they require a bit of work on our end to make sure it works as anticipated. I like option 2 in that it solves the issue mentioned above and makes it more portable and flexible which is exactly what we are looking for. Thanks for the response to the feature request! |
hi @Bryan-Meier - I've been looking into this a bit more and it's raised some interesting questions. I've started a discussion here where I'd be happy to hear any thoughts you have on this! The most important question I'd have for you with respect to this issue is, which of the following do you like better? 🙂 # say we saved some class earlier
# class CustomBlock(Block):
# some_value: Any
# and later, we want to load a view of this block
from prefect.blocks.core import BlockView
block_view_A: BlockView = BlockView.from_block_name("custom-block-type/my-block-name")
print(block_view.some_value) # we could have attribute access here via `extra="allow"`
# or perhaps instead the view is just the raw contents of the block document
from prefect.blocks.core import get_block_view
block_view_B: dict = get_block_view("custom-block-type/my-block-name")
print(block_view.get("some_value")) I understand it may be the case that a thanks for your feedback thus far! |
Hi @zzstoatzz, Thanks for reaching out and asking my opinion. Honestly, either would work for us but I think option 1 presents a cleaner solution where attributes become first class to object. So, my preference would be block_view_A. The biggest thing for us is just being able to specify the data type for the attribute. (i.e. SecretStr, etc.) Thanks again! |
thanks for the feedback @Bryan-Meier! just to keep you updated, I've opened this PR which we should be able to move along once we fix an inconsistency in the block form in the UI, which I imagine would be useful to you |
Describe the current behavior
There is currently no generic block that can be used to store user defined keys can values in a secure way. All the current blocks are system specific, which may not match or work with home grown systems that the user already has in place.
Describe the proposed behavior
It would be great to have a generic block type baked into Prefect that would allow the flexibility to store generic attributes and values like username and password which can then be generically used within Prefect code. I could imagine the block type being a SecretDict that stores it's values as Dict(str, SecretStr).
Example Use
An example would be for SharePoint Credentials:
Another example maybe for a password protected zip file:
Additional context
No response
The text was updated successfully, but these errors were encountered: