Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

More efficient s3 paging #2780

Merged
merged 8 commits into from
Jun 16, 2017
Merged

More efficient s3 paging #2780

merged 8 commits into from
Jun 16, 2017

Commits on May 30, 2017

  1. Move appendIfMissing function to common location. 

    The appendIfMissing function is used by several physical backends so
move the function to a common file.
    @deverton
    deverton committed May 30, 2017
    Configuration menu
    Copy the full SHA
    c16e024 View commit details
    Browse the repository at this point in the history

  2. Have S3 group folders before sending response. 

    The ListObjectsV2 API allows you to specify a folder delimiter when
listing objects in a bucket. This makes S3 group the folders together
and only list the objects in the immediate prefix.

For prefixes with a large number of objects underneath, this can speed
up querying of intermediate paths significantly.
    @deverton
    deverton committed May 30, 2017
    Configuration menu
    Copy the full SHA
    a81736e View commit details
    Browse the repository at this point in the history

  3. Allow S3 integration tests to use shared creds. 

    This allows you to run the S3 integration tests with the
~/.aws/credentials file and AWS_PROFILE environment variable.
    @deverton
    deverton committed May 30, 2017
    Configuration menu
    Copy the full SHA
    8f58b3d View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2017

  1. Merge remote-tracking branch 'origin' into more-efficient-s3-paging

    @deverton
    deverton committed Jun 5, 2017
    Configuration menu
    Copy the full SHA
    7d2c3f1 View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2017

  1. Use the awsutil helper to find test creds

    @deverton
    deverton committed Jun 6, 2017
    Configuration menu
    Copy the full SHA
    9dde160 View commit details
    Browse the repository at this point in the history

  2. Move appendIfMissing to helper/strutil

    @deverton
    deverton committed Jun 6, 2017
    Configuration menu
    Copy the full SHA
    a51865c View commit details
    Browse the repository at this point in the history

  3. Check we can get credentials before starting test.

    @deverton
    deverton committed Jun 6, 2017
    Configuration menu
    Copy the full SHA
    9b6b9c1 View commit details
    Browse the repository at this point in the history

  4. Typo fix.

    @deverton
    deverton committed Jun 6, 2017
    Configuration menu
    Copy the full SHA
    9c1935e View commit details
    Browse the repository at this point in the history