Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to download from S3 buckets #671

Merged

Conversation

danielmitterdorfer
Copy link
Member

With this commit we add the ability to download from S3 buckets. A
download URL must start with the scheme s3:// to be recognized as S3
bucket. Provided that S3 access is properly configured Rally can also
handle downloads from private S3 buckets.

With this commit we add the ability to download from S3 buckets. A
download URL must start  with the scheme `s3://` to be recognized as S3
bucket. Provided that S3 access is properly configured Rally can also
handle downloads from private S3 buckets.
@danielmitterdorfer danielmitterdorfer added enhancement Improves the status quo :Track Management New operations, changes in the track format, track download changes and the like :Benchmark Candidate Management Anything affecting how Rally sets up Elasticsearch labels Mar 29, 2019
@danielmitterdorfer danielmitterdorfer added this to the 1.1.0 milestone Mar 29, 2019
return expected_size_in_bytes


def download_http(url, local_path, expected_size_in_bytes=None, progress_indicator=None):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does it also make sense to add a unit test for download_http? I couldn't find if we have a test for download().. although it is a well executed code path so not sure if we need it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tbh, I attempted to do it but I ended up mocking so much code that I had the impression it's not really worth it.

Copy link
Contributor

@ebadyano ebadyano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thank you!

@danielmitterdorfer danielmitterdorfer merged commit 4adf0ec into elastic:master Apr 1, 2019
@danielmitterdorfer danielmitterdorfer deleted the add-s3-support branch April 1, 2019 11:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Benchmark Candidate Management Anything affecting how Rally sets up Elasticsearch enhancement Improves the status quo :Track Management New operations, changes in the track format, track download changes and the like
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants