Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DynamoDB] - Allow passing ddb endpoint_url to enable feast local testing #2358

Closed
vlin-lgtm opened this issue Mar 3, 2022 · 3 comments · Fixed by #2485
Closed

[DynamoDB] - Allow passing ddb endpoint_url to enable feast local testing #2358

vlin-lgtm opened this issue Mar 3, 2022 · 3 comments · Fixed by #2485
Assignees
Labels
good first issue Good for newcomers kind/feature New feature or request

Comments

@vlin-lgtm
Copy link

Is your feature request related to a problem? Please describe.
Currently in feature_store.yaml, we can only specify a region for DynamoDB provider. As a result, it requires an actual DynamoDB to be available when we want to do local development/testing or integration testing in a sandbox environment.

Describe the solution you'd like
A way to solve this is to let user pass an endpoint_url. More information can be found here.
This way, users can install and run a local dynamoDB, and use it as an online store locally.
The is especially useful when:

  • accessing real DynamoDB requires a tedious and time-consuming steps (e.g.: IAM role set up, permissions, etc.) and these steps can be deferred later,
  • integration testing locally, in docker, in Jenkins

Describe alternatives you've considered
N/A

Additional context
Not much but the initial slack thread can be found here, which was followed by a slack message from our team member
image

@vlin-lgtm vlin-lgtm added the kind/feature New feature or request label Mar 3, 2022
@achals achals added the good first issue Good for newcomers label Mar 4, 2022
@TremaMiguel
Copy link
Member

@vlin-lgtm I think this one is kind of easy _initialize_dynamodb_client and _initialize_dynamodb_resource would now accept endpoint_url

def _initialize_dynamodb_client(region: str, url: str):
    return boto3.client("dynamodb", endpoint_url=url, region_name=region)

which can be passed through the DynamoDBOnlineStoreConfig.

But, I'm not sure how will this be for local integration tests, as far as I know just need to change the DYNAMODB_CONFIG with an additional parameter endpoint_url and set an IntegrationTestConfig with DynamoDB as Online Store.

What are your thoughts?

P.D. I'm happy to contribute with this

@vlin-lgtm
Copy link
Author

Thanks so much, @TremaMiguel!

But, I'm not sure how will this be for local integration tests, ....

My understanding is one can run a local dynamoDB on their development machine or in a docker container, and by specifying an endpoint_url to for example localhost:8080, all DynamoDB invocations will go to the localhost instance instead of an actual one in AWS.

This is useful for integration testing as everything is still "local".

Would love it if you can help contribute to this 🙏

@TremaMiguel
Copy link
Member

@vlin-lgtm Thanks for replying

I agree with the local integration tests, additionaly I think this development could help with #2400

I'll start working on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers kind/feature New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants