Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generic Secure Block #14899

Closed
Bryan-Meier opened this issue Aug 12, 2024 · 5 comments · Fixed by #14980
Closed

Generic Secure Block #14899

Bryan-Meier opened this issue Aug 12, 2024 · 5 comments · Fixed by #14980
Labels
enhancement An improvement of an existing feature

Comments

@Bryan-Meier
Copy link

Describe the current behavior

There is currently no generic block that can be used to store user defined keys can values in a secure way. All the current blocks are system specific, which may not match or work with home grown systems that the user already has in place.

Describe the proposed behavior

It would be great to have a generic block type baked into Prefect that would allow the flexibility to store generic attributes and values like username and password which can then be generically used within Prefect code. I could imagine the block type being a SecretDict that stores it's values as Dict(str, SecretStr).

Example Use

An example would be for SharePoint Credentials:

{
    'url': 'x.sharepoint.com/sites/home',
    'username': 'some_user',
    'password': 'some_password'
}

Another example maybe for a password protected zip file:

{
    'password': 'some_zip_password'
}

Additional context

No response

@Bryan-Meier Bryan-Meier added the enhancement An improvement of an existing feature label Aug 12, 2024
@cicdw
Copy link
Member

cicdw commented Aug 13, 2024

Hey @Bryan-Meier and thank you for the request! It makes sense to me. For my own understanding I'd like to offer up two alternatives and see if either of them appeal to you - either way I'm happy to support a SecretDict type Block, this is more to better understand patterns around Blocks that we can improve.

Option 1: Custom Class

You could create a new Block type yourself, e.g.,

from prefect.blocks.core import Block
from pydantic import SecretStr


class MySecretDict(Block):
    username: SecretStr
    password: SecretStr
    url: SecretStr

my_block = MySecretDict(username="foo", password="bar", url="blah")
my_block.save("example")

I'm guessing this isn't ideal because you then need to have the importable class anywhere in which you want to use this block document.

Option 2: ignoring block methods on load

This option is not currently implemented so is more of a thought experiment to see if it would extend the pattern here and resolve other forms of frustration. The main idea is to reference blocks purely as typed-documents and ignore the business logic methods that tend to be on many of our blocks.

First you would still register a new block type / document with the API as above. The big difference would be that loading this block would not require any imports, so you wouldn't need to redefine the class anywhere:

from prefect.blocks.core import Block, inject_block_document

# this block would load the fields as a _pydantic model_ but not attempt to import the 
# class itself
my_block = Block.load("mysecretdict/example", document_only=True)


# this could be extended to a decorator that injects fields into custom classes
@inject_block_document("mysecretdict/example")
class MyRandomClass:
    pass


# would load the fields from the API and inject them as attrs onto this class instance
my_class = MyRandomClass() 

At your convenience I'm curious for your thoughts (or anyone else lurking!) on either of those approaches.

@Bryan-Meier
Copy link
Author

Hi @cicdw,

The comment you made at the end of option 1 is exactly what we are running into. We want to create the block type and then use it anywhere we desire without having to copy the definition of the block into every Prefect job that we plan to use it in. There are definitely ways to do this and they require a bit of work on our end to make sure it works as anticipated.

I like option 2 in that it solves the issue mentioned above and makes it more portable and flexible which is exactly what we are looking for.

Thanks for the response to the feature request!

@zzstoatzz
Copy link
Collaborator

hi @Bryan-Meier - I've been looking into this a bit more and it's raised some interesting questions.

I've started a discussion here where I'd be happy to hear any thoughts you have on this!

The most important question I'd have for you with respect to this issue is, which of the following do you like better? 🙂

# say we saved some class earlier
# class CustomBlock(Block):
#    some_value: Any

# and later, we want to load a view of this block
from prefect.blocks.core import BlockView

block_view_A: BlockView = BlockView.from_block_name("custom-block-type/my-block-name")
print(block_view.some_value) # we could have attribute access here via `extra="allow"`


# or perhaps instead the view is just the raw contents of the block document
from prefect.blocks.core import get_block_view

block_view_B: dict = get_block_view("custom-block-type/my-block-name")
print(block_view.get("some_value"))

I understand it may be the case that a SecretDictionary block type would solve your specific issue more directly (this still may make sense to do!), but we wanted to take a look at the general problem here.

thanks for your feedback thus far!

@Bryan-Meier
Copy link
Author

Hi @zzstoatzz,

Thanks for reaching out and asking my opinion. Honestly, either would work for us but I think option 1 presents a cleaner solution where attributes become first class to object. So, my preference would be block_view_A. The biggest thing for us is just being able to specify the data type for the attribute. (i.e. SecretStr, etc.)

Thanks again!

@zzstoatzz
Copy link
Collaborator

thanks for the feedback @Bryan-Meier!

just to keep you updated, I've opened this PR which we should be able to move along once we fix an inconsistency in the block form in the UI, which I imagine would be useful to you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement An improvement of an existing feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants