Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix transfer config import #2458

Merged
merged 44 commits into from
Jul 16, 2018
Merged

Fix transfer config import #2458

Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
652c706
AWS_SESSION_Token read from ENV
ouanixi May 20, 2017
25795e1
AWS_SESSION_Token read from ENV
ouanixi May 20, 2017
5f65398
add s3client
Oct 16, 2017
8301512
s3client using boto3
Oct 16, 2017
bbb4359
removed AWS creds from travis and tox env vars
Oct 16, 2017
79cb0f4
removed AWS creds from travis and tox env vars
Oct 16, 2017
825adf4
importing form contrib.s3 instead of s3 directly
Oct 16, 2017
0cdce3e
AWS credentials passed as env vars
Oct 16, 2017
d74d179
fixed wrong s3 import
Oct 16, 2017
04199b1
redshift tests passing locally
Oct 16, 2017
edbc1de
test_ecs mocked
Oct 16, 2017
776f25d
changed all byte instances to strings
Oct 16, 2017
6073868
writing to file as bytes
Oct 16, 2017
bcce526
decoding bytes to strings for python3 tests
Oct 16, 2017
b9fca9a
contrib/redshift test reading creds from env vars
Oct 16, 2017
00b2bb5
travis build on master
Oct 16, 2017
4e00b55
flake8 redshift_test
Oct 16, 2017
b70aad8
reverted to master redshift_test
Oct 17, 2017
574022f
skipping test/redshift_test.py on travis
Oct 17, 2017
363e64d
flake8 test/redshift_test.py
Oct 17, 2017
24df402
flake8 test/redshift_test.py
Oct 17, 2017
20d83c7
skipping test/redshift_test.py on travis
Oct 17, 2017
21e8af8
moved test/redshift_test.py to test/contrib/
Oct 17, 2017
848d555
redshift_test using mocked S3Target open
Oct 17, 2017
2de69b7
redshift_test using mocked S3Target open
Oct 17, 2017
c714016
redshift_test using mocked S3Target open
Oct 17, 2017
0eb8d2f
moved redshift_test to contrib folder and refactored redshiftmanifest…
Oct 17, 2017
5a3e9d3
fixed flake8 errors
Oct 17, 2017
025c72c
fixed flake8 errors
Oct 17, 2017
03ad80b
added multipart s3 tests
Oct 18, 2017
81b91a5
added multipart s3 tests
Oct 18, 2017
4db2c2b
added multipart s3 tests
Oct 18, 2017
835b305
travis to trigger builds on master
Oct 18, 2017
06cce0f
skip on travis message made clearer
ouanixi Nov 22, 2017
ed2a736
add s3client with rebase
ouanixi Nov 22, 2017
35e5a73
Merge branch 's3client'
ouanixi Nov 22, 2017
9ce65f4
Merge branch 'master' into s3client
ouanixi Nov 22, 2017
21c4b3b
removed arn resolution
Nov 23, 2017
3bb1f1e
s3client resolving auth alone
Nov 23, 2017
038833c
logging boto3 resource call
ouanixi Nov 23, 2017
c779fb8
merge upstream
ouanixi Feb 2, 2018
e461412
master pull
Jul 13, 2018
c787568
import TransferConfig instead of full boto3 module
Jul 13, 2018
73dc7ae
revert to travis build on master
Jul 13, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 4 additions & 6 deletions luigi/contrib/s3.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -283,11 +283,10 @@ def put_multipart(self, local_path, destination_s3_path, part_size=8388608, **kw
raise DeprecatedBotoClientException(
'encrypt_key deprecated in boto3. Please refer to boto3 documentation for encryption details.')

import boto3
from boto3.s3.transfer import TransferConfig
# default part size for boto3 is 8Mb, changing it to fit part_size
# provided as a parameter
transfer_config = boto3.s3.transfer.TransferConfig(
multipart_chunksize=part_size)
transfer_config = TransferConfig(multipart_chunksize=part_size)

(bucket, key) = self._path_to_bucket_and_key(destination_s3_path)

Expand Down Expand Up @@ -318,10 +317,9 @@ def copy(self, source_path, destination_path, threads=100, start_time=None, end_

# don't allow threads to be less than 3
threads = 3 if threads < 3 else threads
import boto3
from boto3.s3.transfer import TransferConfig

transfer_config = boto3.s3.transfer.TransferConfig(
max_concurrency=threads, multipart_chunksize=part_size)
transfer_config = TransferConfig(max_concurrency=threads, multipart_chunksize=part_size)
total_keys = 0

if self.isdir(source_path):
Expand Down