Add config variables for changing S3 runtime values

[jamesls]

This pull requests adds support for the following parameters in the ~/.aws/config file:

• max_concurrent_requests: The maximum number of concurrent transfer requests to S3 at any given moment.
• max_queue_size: The maximum number of transfer tasks queued at any given point in time.
• multipart_threshold: The size at which we switch to multipart uploads/downlaods. The appropriate values are specified below.
• multipart_chunk_size: The chunk size to use when using multipart transfers.

A sample config looks like:

[profile development]
aws_access_key_id = foo
aws_secret_access_key = bar
region = us-west-2
s3 =
  max_concurrent_requests = 20
  max_queue_size = 10000
  multipart_threshold = 128MB
  multipart_chunk_size = 64MB

This addresses several needs we've heard including:

• The ability to limit the number of parallel s3 transfers (Limit number of parallel s3 transfers #907, and indirectly Add ability to limit bandwidth for S3 uploads/downloads #1090)
• The ability to set the queue size for better progress info: Expose max queue size in S3 as a config/param #699
• Add the ability to set the threshold at which we switch to multipart operations
• Set the part size to use for multipart operations

This still needs documentation, but the code is ready for review.

Code Overview

While working on this code, there were parts of the code I updated before making any functional changes. These include things like pep8 formatting, removing code that's no longer called, improving some of the semantics of the CommandParameters class. None of these changes have any functional changes:

fb51db1 - Cleanup code formatting
60f7377 - Remove unused method
ab7ba1a - Remove session dep from CommandParameters
e9fdaf7 - Remove trailing/leading whitespace
2e3477d - Move part size constants out to their respective modules
9ba4d38 - Move constants to transferconfig

Then the two commits that add in the change. The first is the addition of a RuntimeConfig object that manages all of the merging/converting of config file values:

697662b - Add class for merging converting s3 runtime config

Then plumbing this newly created class into the S3 command codebase:

db24830 - Plumb through S3 runtime config

I'd suggest taking a look at the last two commits on their own and then reviewing the rest.

cc @kyleknap @danielgtaylor

[coveralls]

Coverage decreased (-0.12%) to 91.63% when pulling db24830 on jamesls:s3-configuration into 86f7f52 on aws:develop.

[coveralls]

Coverage increased (+0.02%) to 91.77% when pulling db24830 on jamesls:s3-configuration into 86f7f52 on aws:develop.

[jamesls]

Looking at the coveralls coverage drop...

[jamesls]

Also, just noticed that there's an integ test failure for streaming. Looks like I missed plumbing this into streaming s3handler as well, which I believe should not use the runtime config settings.

[kyleknap]

Shouldn't this be

num_threads=self._runtime_config['max_concurrent_requests']-1

Because the main thread is acting as a producer by calling the ListObjects operation.

[jamesls]

After initially implementing that, I ended up making this just match the number of threads. The main reason is that if we do add max - 1, then the minimum value needs to be 2, which I think would confuse customers. It's also not intuitive that you can't set this value to 1. If a user wants to "turn off" parallelism, you'd think you'd set max_concurrent_requests to 1, but they would need to set max_concurrent_requests to 2. I could see us getting questions about that.

Another option is to rename this to be more explicit. If we called it max_transfer_requests maybe that would clear up what this is doing. Although I think max_concurrent_requests is more clear than max_transfer_threads. What do you think?

Or another option would be to be clear in our documentation what this means. It's the number of concurrent requests uploading/downloading/copying to s3. It does not include any ListObjects calls.

[kyleknap]

Yeah I like that last option the best. If I had no knowledge of the internals of the s3 commands, I would also find it confusing that the lowest that you can make it is 2. I do not like max_transfer_requests because it is ambiguous (it seems to imply that we will at most do that maximum amount of transfers for the one command).

[kyleknap]

Looks good. I guess my main comment was related to the interpretation of max_concurrent_requests. I just want to be on the same page with you before it gets merged.

[coveralls]

Coverage increased (+0.16%) to 91.9% when pulling 31260e1 on jamesls:s3-configuration into 86f7f52 on aws:develop.

[coveralls]

Coverage increased (+0.15%) to 91.89% when pulling 57f4937 on jamesls:s3-configuration into 86f7f52 on aws:develop.

[jamesls]

@kyleknap I believe I've incorporated all the feedback. Coveralls also looks good now. Let me know if I've missed anything.

cc @danielgtaylor

[coveralls]

Coverage increased (+0.15%) to 91.9% when pulling 4d44ce6 on jamesls:s3-configuration into 86f7f52 on aws:develop.

[kyleknap]

Thanks! 🚢

[jamesls]

Closing, this was merged via: 9911da3

[joelvanvelden]

@jamesls,
The given sample config has an error:
"multipart_chunk_size = 64MB"
should instead read:
multipart_chunksize = 64MB

I found this out the hard way :(

Also, this (very useful) new feature is not documented anywhere apart from this pull request.

[jamesls]

@joelvanvelden Thanks for pointing that out. I wish I would have named it multipart_chunk_size to be consistent with max_queue_size but I suppose it's too late to change.

Regarding the documentation, there's an existing pull request out (#1147) that adds topic guides to the CLI help system (as well as the online html docs). This gives us a place to document advanced topics such as these s3 settings. We'll be adding a topic guide on s3 configuration shortly.

[TimLudwinski]

Unfortunately, setting max_concurrent_requests doesn't solve the issue of limiting the bandwidth for me. With 1 connection (I verified that there was only one though netstat), our 100Mb pipe is clogged. (Not that having this bandwidth is a bad thing necessarily). We still need #1090 to be implemented.