Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add local testing support for S3 code #5064

Closed
malhotrashivam opened this issue Jan 23, 2024 · 3 comments
Closed

Add local testing support for S3 code #5064

malhotrashivam opened this issue Jan 23, 2024 · 3 comments
Assignees
Labels
feature request New feature or request parquet Related to the Parquet integration s3

Comments

@malhotrashivam
Copy link
Contributor

malhotrashivam commented Jan 23, 2024

In the current code, we have added unit tests for fetching files from remote S3 buckets and have disabled them in our CI jobs to avoid failures. A better way to test this code is to use a local S3 stack and unit test against it.
https://java.testcontainers.org/modules/localstack/ could be one approach to do this.

Make sure to add both python and java tests for this code path.

Found during #4972

@malhotrashivam
Copy link
Contributor Author

@devinrsmith has shared a sample commit here: malhotrashivam#1

@malhotrashivam
Copy link
Contributor Author

https://java.testcontainers.org/modules/minio/ is another good candidate for integration testing.

@devinrsmith
Copy link
Member

Fixed by #5137

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request parquet Related to the Parquet integration s3
Projects
None yet
Development

No branches or pull requests

2 participants