s3 sync broken in 1.14.0 - unexpected keyword argument 'max_bandwidth' #3007

abrom · 2017-11-30T02:58:28Z

> aws s3 sync . s3://<bucket>/<path> --delete

__init__() got an unexpected keyword argument 'max_bandwidth'

Was working fine with 1.13.0 yesterday.

Installation via bundled installer method (awscli-bundle.zip)

Note, we do not configure any defaults via file or CLI. Only args passed through are secret access key, access key id and default region via ENV.

Likely related to #2997

HTH

The text was updated successfully, but these errors were encountered:

hafizramly · 2017-11-30T08:28:55Z

I had the same issue with s3 sync

My version :
aws-cli/1.14.0 Python/2.6.5 Linux/2.6.32-377-ec2 botocore/1.8.4

Step to produce :

aws s3 sync s3://bucket/folder/ /local/folder/ --region us-east-1 --only-show-errors

__init__() got an unexpected keyword argument 'max_bandwidth'

joguSD · 2017-11-30T17:16:49Z

This has been fixed in v1.14.1. Updating again should fix it.

Duplicate: #3006

kelleydon · 2017-11-30T21:22:47Z

I upgraded to v1.14.2 and I'm still getting the error with aws cp.

Steps to produce:

aws s3 cp --region us-east-1 s3://file_location file_location

joguSD · 2017-11-30T21:29:51Z

@kelleydon
I think the upgrade behavior is going to depend on the version of pip, if you're still running into the issue manually update s3transfer by doing:

pip install --upgrade s3transfer

hafizramly · 2017-12-04T03:42:13Z

Its now working correctly after upgrading to the newer version

aws-cli/1.14.2 Python/2.7.6 Linux/3.13.0-135-generic botocore/1.8.6

rflaperuta · 2017-12-05T12:30:43Z

Just got to this problem, here follows the log with --debug option:

2017-12-05 10:21:14,376 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.14.3 Python/3.6.3 Linux/4.10.0-40-generic botocore/1.8.7

2017-12-05 10:21:14,376 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'sync', 's3://ns-coletor-geral-prd/', '.', '--region', 'us-west-2', '--include=/log-inf
ra/coletorws/20171205/*.log', '--debug']
2017-12-05 10:21:14,376 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x7f9f14103378>
2017-12-05 10:21:14,377 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-12-05 10:21:14,377 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7f9f14b701e0>
2017-12-05 10:21:14,377 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-12-05 10:21:14,377 - MainThread - botocore.session - DEBUG - Loading variable credentials_file from defaults.
2017-12-05 10:21:14,377 - MainThread - botocore.session - DEBUG - Loading variable config_file from defaults.
2017-12-05 10:21:14,377 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-12-05 10:21:14,377 - MainThread - botocore.session - DEBUG - Loading variable metadata_service_timeout from defaults.
2017-12-05 10:21:14,377 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-12-05 10:21:14,377 - MainThread - botocore.session - DEBUG - Loading variable metadata_service_num_attempts from defaults.
2017-12-05 10:21:14,377 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-12-05 10:21:14,377 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7f9f17427d90>
2017-12-05 10:21:14,377 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-12-05 10:21:14,377 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x7f9f14109620>
2017-12-05 10:21:14,378 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.s3.anonymous: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,378 - MainThread - botocore.hooks - DEBUG - Event building-command-table.sync: calling handler <function register_sync_strategies at 0x7f9f140ffa60>
2017-12-05 10:21:14,378 - MainThread - botocore.hooks - DEBUG - Event building-command-table.sync: calling handler <function add_waiters at 0x7f9f14109620>
2017-12-05 10:21:14,378 - MainThread - botocore.hooks - DEBUG - Event building-arg-table.sync: calling handler <bound method BaseSync.add_sync_argument of <awscli.customizations.s3.syncs
trategy.sizeonly.SizeOnlySync object at 0x7f9f14041b00>>
2017-12-05 10:21:14,379 - MainThread - botocore.hooks - DEBUG - Event building-arg-table.sync: calling handler <bound method BaseSync.add_sync_argument of <awscli.customizations.s3.syncs
trategy.exacttimestamps.ExactTimestampsSync object at 0x7f9f14041be0>>
2017-12-05 10:21:14,379 - MainThread - botocore.hooks - DEBUG - Event building-arg-table.sync: calling handler <bound method BaseSync.add_sync_argument of <awscli.customizations.s3.syncs
trategy.delete.DeleteSync object at 0x7f9f14041cc0>>
2017-12-05 10:21:14,380 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.paths: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,380 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.dryrun: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,380 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.sync: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f9f14bc25f8>
2017-12-05 10:21:14,380 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.quiet: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,380 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.sync: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f9f14bc25f8>
2017-12-05 10:21:14,380 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.anonymous: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,381 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.acl: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,381 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.follow-symlinks: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,381 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.sync: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f9f14bc25f8>
2017-12-05 10:21:14,381 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.anonymous: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,381 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.sse: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,381 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.sse-c: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,381 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.sse-c-key: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,381 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.sse-kms-key-id: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,381 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.sse-c-copy-source: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,381 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.sse-c-copy-source-key: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,381 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.storage-class: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,381 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.grants: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,381 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.website-redirect: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,381 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.content-type: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,381 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.cache-control: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,381 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.content-disposition: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,381 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.content-encoding: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,381 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.content-language: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,381 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.expires: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,382 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.source-region: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,382 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.only-show-errors: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,382 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.sync: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f9f14bc25f8>
2017-12-05 10:21:14,382 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.anonymous: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,382 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.page-size: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,382 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.ignore-glacier-warnings: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,382 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.sync: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f9f14bc25f8>
2017-12-05 10:21:14,382 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.force-glacier-transfer: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,382 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.sync: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f9f14bc25f8>
2017-12-05 10:21:14,382 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.metadata: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,382 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.metadata-directive: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,382 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.size-only: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,382 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.sync: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f9f14bc25f8>
2017-12-05 10:21:14,382 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.exact-timestamps: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,382 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.sync: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f9f14bc25f8>
2017-12-05 10:21:14,382 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.sync.delete: calling handler <function uri_param at 0x7f9f14c017b8>
2017-12-05 10:21:14,382 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.sync: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f9f14bc25f8>
2017-12-05 10:21:14,382 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-12-05 10:21:14,382 - MainThread - botocore.session - DEBUG - Loading variable ca_bundle from defaults.
2017-12-05 10:21:14,382 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-12-05 10:21:14,383 - MainThread - botocore.session - DEBUG - Loading variable api_versions from defaults.
2017-12-05 10:21:14,383 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2017-12-05 10:21:14,383 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2017-12-05 10:21:14,383 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2017-12-05 10:21:14,383 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2017-12-05 10:21:14,383 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python3.6/site-packages/botocore/data/endpoints.json
2017-12-05 10:21:14,385 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-12-05 10:21:14,386 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/lib/python3.6/site-packages/botocore/data/s3/2006-03-01/service-2.json
2017-12-05 10:21:14,392 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f9f15a59158>
2017-12-05 10:21:14,392 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f9f15a4fea0>
2017-12-05 10:21:14,393 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None
2017-12-05 10:21:14,396 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2017-12-05 10:21:14,396 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2017-12-05 10:21:14,397 - MainThread - botocore.client - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.
2017-12-05 10:21:14,397 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-12-05 10:21:14,397 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-12-05 10:21:14,397 - MainThread - botocore.session - DEBUG - Loading variable ca_bundle from defaults.
2017-12-05 10:21:14,397 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-12-05 10:21:14,397 - MainThread - botocore.session - DEBUG - Loading variable api_versions from defaults.
2017-12-05 10:21:14,397 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-12-05 10:21:14,398 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f9f15a59158>
2017-12-05 10:21:14,398 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f9f15a4fea0>
2017-12-05 10:21:14,398 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None
2017-12-05 10:21:14,398 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None
2017-12-05 10:21:14,400 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2017-12-05 10:21:14,400 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2017-12-05 10:21:14,400 - MainThread - botocore.client - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.
2017-12-05 10:21:14,400 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-12-05 10:21:14,400 - MainThread - botocore.session - DEBUG - Loading variable ca_bundle from defaults.
2017-12-05 10:21:14,401 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-12-05 10:21:14,401 - MainThread - botocore.session - DEBUG - Loading variable api_versions from defaults.
2017-12-05 10:21:14,401 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2017-12-05 10:21:14,401 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f9f15a59158>
2017-12-05 10:21:14,401 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f9f15a4fea0>
2017-12-05 10:21:14,401 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None
2017-12-05 10:21:14,415 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2017-12-05 10:21:14,416 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3
2017-12-05 10:21:14,416 - MainThread - botocore.client - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.
2017-12-05 10:21:14,416 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/awscli/clidriver.py", line 207, in main
return command_table[parsed_args.command](remaining, parsed_args)
File "/usr/local/lib/python3.6/site-packages/awscli/customizations/commands.py", line 190, in call
parsed_globals)
File "/usr/local/lib/python3.6/site-packages/awscli/customizations/commands.py", line 187, in call
return self._run_main(parsed_args, parsed_globals)
File "/usr/local/lib/python3.6/site-packages/awscli/customizations/s3/subcommands.py", line 697, in _run_main
return cmd.run()
File "/usr/local/lib/python3.6/site-packages/awscli/customizations/s3/subcommands.py", line 1025, in run
self._client, result_queue)
File "/usr/local/lib/python3.6/site-packages/awscli/customizations/s3/s3handler.py", line 84, in call
self._runtime_config)
File "/usr/local/lib/python3.6/site-packages/awscli/customizations/s3/transferconfig.py", line 119, in create_transfer_config_from_runtime_config
return TransferConfig(**kwargs)
TypeError: init() got an unexpected keyword argument 'max_bandwidth'
2017-12-05 10:21:14,417 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

init() got an unexpected keyword argument 'max_bandwidth'

jamesls · 2017-12-11T17:32:47Z

For anyone that's still running into this issue, make sure you have the latest version of s3transfer installed. This should automatically be handled via pip (our dependency on s3transfer has been updated https://github.com/aws/aws-cli/blob/develop/setup.py#L30), but in case not, please run pip install --upgrade s3transfer.

Let us know if the above steps are still not working for you.

jgramoll · 2017-12-29T15:45:53Z

I am still getting the error.

aws --version
aws-cli/1.14.16 Python/2.7.6 Linux/3.13.0-137-generic botocore/1.8.20

pip show s3transfer
"Version: 0.1.12"

sakers · 2018-01-10T10:32:30Z

I am still getting this error with sn aws s3 sync:

aws s3 sync s3:///app_data/ /data/app_data

init() got an unexpected keyword argument 'max_bandwidth'

aws --version
aws-cli/1.14.22 Python/2.7.5 Linux/3.10.0-693.5.2.el7.x86_64 botocore/1.8.26

pip show s3transfer
Name: s3transfer
Version: 0.1.12

meowtochondria · 2018-04-13T00:24:33Z

just a pointer in case someone lands here - check via your package manager if your OS has previous version of s3transfer installed. It can happen when you used apt or yum (or your distro's package manager) to install awscli. python2-s3transfer (v0.10.0) was still hanging around for me even when i removed awscli using package manager. I had to reinstall s3transfer using pip after removing python2-s3transfer.

karnauskas · 2018-05-11T12:40:55Z

Got same __init__() got an unexpected keyword argument 'max_bandwidth'.

aws --version
aws-cli/1.14.28 Python/2.7.5 Linux/3.10.0-693.5.2.el7.x86_64 botocore/1.8.35

CiliaFred · 2018-05-11T14:52:59Z

Got the same using yum install awscli and CentOS AMI
Fixed it by using pip to install awscli
yum install -y curl python36
curl https://bootstrap.pypa.io/3.3/get-pip.py
python36 get-pip.py
pip install awscli==1.15.19

pbudzon · 2018-05-20T11:43:55Z

aws cp ... is affected as well.

awscli package in Centos base yum repo seems to be broken (this affects not only the AMI).
On "GenericCloud" image removing yum package and installing awscli from pip2 solved the problem.
On AMI had to remove the yum package and install cli from pip3 (pip2 version was also broken).

heraldb · 2018-05-31T18:49:57Z

Indeed, installing awscli via pip worked fine. I used python3, just like CiliaFred

What I did on Centos 7.5:

yum remove awscli python-s3transfer
yum install -y python36 python34-pip
pip3 install awscli==1.15.19

IlariExove · 2018-06-14T14:32:07Z

Fedora EPEL maintainers should be notified to get this fixed in CentOS distros. I'll create a new EPEL Bugzilla ticket about this if I can't find an existing one there.

Co-authored-by: Sriram Madapusi Vasudevan <3770774+sriram-mv@users.noreply.github.com>

* feat: Delete methods for CF stacks and S3 files (aws#2981) * Added methods for cf and s3 files and init UI * Added unit tests for utils methods and s3_uploader * Removed s3_bucket and s3_prefix click options * Fixed lint errors and added few unit tests * Make black happy * Added LOG statements * Added and updated changes based on CR * Fixed the unit tests in artifact_exporter.py * Update HELP_TEXT in delete/command.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Updated code based on Chris' comments * Small changes and fixes based on the comments * Removed region prompt * Update SAM context values for profile and region in delete_context.py * Added typing for get_cf_template_name method * Added stack_name prompt if the stack_name is not present in samconfig file * Replace [] with get() for stack-name in delete_context.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Delete template artifacts (aws#3022) * Added methods for cf and s3 files and init UI * Added unit tests for utils methods and s3_uploader * Removed s3_bucket and s3_prefix click options * chore: Increase awareness of same file warning during package (aws#2946) * chore: increase awareness of same file warning during package * fix formatting & grammar Co-authored-by: Mathieu Grandis <73313235+mgrandis@users.noreply.github.com> * fix: Allow the base64Encoded field in REST Api, skip validation of unknown fields and validate missing statusCode for Http Api (aws#2941) * fix API Gateway emulator: - skip validating the non allowed fields for Http Api Gateway, as it always skip the unknown fields - add base64Encoded as an allowed field for Rest Api gateway - base64 decoding will be always done for Http API gateway if the lambda response isBase64Encoded is true regardless the content-type - validate if statusCode is missing in case of Http API, and payload version 1.0 * - accept "true", "True", "false", "False" as valid isBase64Encoded values. - Validate on other isBase64Encoded Values - add more integration && unit test cases * fix lint && black issues * use smaller image to test Base64 response * Fixed lint errors and added few unit tests * Make black happy * Added methods for deleting template artifacts * Wait method added for delete cf api * Added LOG statements * Added and updated changes based on CR * Fixed the unit tests in artifact_exporter.py * Update HELP_TEXT in delete/command.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Updated code based on Chris' comments * Added condition for resources that have deletionpolicy specified * Small changes and fixes based on the comments * Removed region prompt * Added unit tests for ecr delete method and typing for methods * Reformatted delete_context and added option to skip user prompts * Removed return type from artifact_exporter for delete method * Added unit tests for artifact_exporter and delete_context * Added more unit tests for delete_context and artifact_exporter * Added more unit tests for delete_context and artifact_exporter * Added docs and comments for artifact_exporter and ecr_uploader * Added log statements in delete_context and some updates in unit tests * Changed force to no-prompts and updated ecr delete method error handling * Created a separate function for parsing ecr url in ecr_uploader * Reformatted Template class init to pass template_str and init template_dict * Changed how s3 url is obtained for resource_zip edge-case: aws:glue:job * Fixed edge case where resource artifact points to a path style url * run Make black * Made the parse s3 url funcs protected and defined a parent method and modified delete method for ResourceImageDict * Changed parse_ecr_url function name to parse_image_url Co-authored-by: Mehmet Nuri Deveci <5735811+mndeveci@users.noreply.github.com> Co-authored-by: Mathieu Grandis <73313235+mgrandis@users.noreply.github.com> Co-authored-by: Mohamed Elasmar <71043312+moelasmar@users.noreply.github.com> Co-authored-by: Chris Rehn <crehn@outlook.com> * Get s3 info cf template (aws#3050) * Added methods for cf and s3 files and init UI * Added unit tests for utils methods and s3_uploader * Removed s3_bucket and s3_prefix click options * chore: Increase awareness of same file warning during package (aws#2946) * chore: increase awareness of same file warning during package * fix formatting & grammar Co-authored-by: Mathieu Grandis <73313235+mgrandis@users.noreply.github.com> * fix: Allow the base64Encoded field in REST Api, skip validation of unknown fields and validate missing statusCode for Http Api (aws#2941) * fix API Gateway emulator: - skip validating the non allowed fields for Http Api Gateway, as it always skip the unknown fields - add base64Encoded as an allowed field for Rest Api gateway - base64 decoding will be always done for Http API gateway if the lambda response isBase64Encoded is true regardless the content-type - validate if statusCode is missing in case of Http API, and payload version 1.0 * - accept "true", "True", "false", "False" as valid isBase64Encoded values. - Validate on other isBase64Encoded Values - add more integration && unit test cases * fix lint && black issues * use smaller image to test Base64 response * Fixed lint errors and added few unit tests * Make black happy * Added methods for deleting template artifacts * Wait method added for delete cf api * Added LOG statements * Added and updated changes based on CR * Fixed the unit tests in artifact_exporter.py * Update HELP_TEXT in delete/command.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Updated code based on Chris' comments * Added condition for resources that have deletionpolicy specified * Small changes and fixes based on the comments * Removed region prompt * Added unit tests for ecr delete method and typing for methods * Reformatted delete_context and added option to skip user prompts * Removed return type from artifact_exporter for delete method * Added unit tests for artifact_exporter and delete_context * Added more unit tests for delete_context and artifact_exporter * Added more unit tests for delete_context and artifact_exporter * Added docs and comments for artifact_exporter and ecr_uploader * Added log statements in delete_context and some updates in unit tests * Changed force to no-prompts and updated ecr delete method error handling * Created a separate function for parsing ecr url in ecr_uploader * Reformatted Template class init to pass template_str and init template_dict * Changed how s3 url is obtained for resource_zip edge-case: aws:glue:job * Fixed edge case where resource artifact points to a path style url * run Make black * Made the parse s3 url funcs protected and defined a parent method and modified delete method for ResourceImageDict * Added methods to extract s3 info from cf template * Added testing for get_s3_info method for artifact_exporter and s3_uploader methods * Removed commented code and updated method docstring * Better error handling for s3 delete artifacts and fixed bug for getting s3 resources information * Changed get_s3_info to get_property_value and changed output text for s3 delete method Co-authored-by: Mehmet Nuri Deveci <5735811+mndeveci@users.noreply.github.com> Co-authored-by: Mathieu Grandis <73313235+mgrandis@users.noreply.github.com> Co-authored-by: Mohamed Elasmar <71043312+moelasmar@users.noreply.github.com> Co-authored-by: Chris Rehn <crehn@outlook.com> * Sam delete integration testing (aws#3076) * Added methods for cf and s3 files and init UI * Added unit tests for utils methods and s3_uploader * Removed s3_bucket and s3_prefix click options * chore: Increase awareness of same file warning during package (aws#2946) * chore: increase awareness of same file warning during package * fix formatting & grammar Co-authored-by: Mathieu Grandis <73313235+mgrandis@users.noreply.github.com> * fix: Allow the base64Encoded field in REST Api, skip validation of unknown fields and validate missing statusCode for Http Api (aws#2941) * fix API Gateway emulator: - skip validating the non allowed fields for Http Api Gateway, as it always skip the unknown fields - add base64Encoded as an allowed field for Rest Api gateway - base64 decoding will be always done for Http API gateway if the lambda response isBase64Encoded is true regardless the content-type - validate if statusCode is missing in case of Http API, and payload version 1.0 * - accept "true", "True", "false", "False" as valid isBase64Encoded values. - Validate on other isBase64Encoded Values - add more integration && unit test cases * fix lint && black issues * use smaller image to test Base64 response * Fixed lint errors and added few unit tests * Make black happy * Added methods for deleting template artifacts * Wait method added for delete cf api * Added LOG statements * Added and updated changes based on CR * Fixed the unit tests in artifact_exporter.py * Update HELP_TEXT in delete/command.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Updated code based on Chris' comments * Added condition for resources that have deletionpolicy specified * Small changes and fixes based on the comments * Removed region prompt * Added unit tests for ecr delete method and typing for methods * Reformatted delete_context and added option to skip user prompts * Removed return type from artifact_exporter for delete method * Added unit tests for artifact_exporter and delete_context * Added more unit tests for delete_context and artifact_exporter * Added more unit tests for delete_context and artifact_exporter * Added docs and comments for artifact_exporter and ecr_uploader * Added log statements in delete_context and some updates in unit tests * Init integration tests methods for sam delete * Added more template files as a list for sam delete integration testing * Changed force to no-prompts and updated ecr delete method error handling * Created a separate function for parsing ecr url in ecr_uploader * Reformatted Template class init to pass template_str and init template_dict * Changed how s3 url is obtained for resource_zip edge-case: aws:glue:job * Fixed edge case where resource artifact points to a path style url * run Make black * Added 2 more integrations tests no_stack_deployed and delete for image type resources * Made the parse s3 url funcs protected and defined a parent method and modified delete method for ResourceImageDict * Added methods to extract s3 info from cf template * Added testing for get_s3_info method for artifact_exporter and s3_uploader methods * Added few more integration tests for delete * Merged delete-template-artifacts branch code and updated this branch code * Added tests for no prefix present for zip and image * Added a check to confirm input stack is deleted for all the integration tests * Removed commented code and updated method docstring * Better error handling for s3 delete artifacts and fixed bug for getting s3 resources information * Changed get_s3_info to get_property_value and changed output text for s3 delete method * Added integration test for deleting nested stacks Co-authored-by: Mehmet Nuri Deveci <5735811+mndeveci@users.noreply.github.com> Co-authored-by: Mathieu Grandis <73313235+mgrandis@users.noreply.github.com> Co-authored-by: Mohamed Elasmar <71043312+moelasmar@users.noreply.github.com> Co-authored-by: Chris Rehn <crehn@outlook.com> * Auto ECR Companion Stack Deletion (aws#3080) * Added ecr_bootstrap * Added companion_stack_manager * Added Companion Stack Manager * Added update_companion_stack * Updated companion_stack_builder File Name * Formatted with Black * Updated get_unreferenced_repos * Updated guided_context to Use Companion Stack * Added Delete Auto Create ECR Repo Prompt * Updated prompt_image_repository Flow * Added --resolve-image-repos * Addressed Some of Pylint Issues * Updated Helper Text * Updated Comments * Fixed Typing * Removed Unused Imports * Updated Unit Tests * Updated UX and Fixed Windows ANSI * Updated Unit Tests * Fixed Import Order * Added Ignore Import Check * Added Integration Tests * Updated help text. Co-authored-by: Chris Rehn <crehn@outlook.com> * Added Comments for Name Generation * Updated Image Option Validator * Updated CompanionStackBuilder to Use Dict instead of String * Fixed Argument Ordering * Added Mapping Information to Help Text * Updated delete_unreferenced_repos Doc String * Updated sync_repos Doc String * Added Justification for ECR Repo Physical ID * Refactored to be Less Coupled * Refactored for prompt_specify_repos * Fixed Unit Test * Moved WaiterConfig Out of Methods * Updated Typing * Updated Managed S3 Template to be Dict * Fixed Typo * Added Comments for _save_image_repositories * Fixed Pylint Issue * Added Missing Check for unreferenced_repo_uris * Updated Variable Name * Fixed Typo * Updated Windows Check to Use platform.system() * Updated update_companion_stack Logic * Fixed Comment Typo * Fixed Typos * Fixed Test Name * Added methods for cf and s3 files and init UI * Added unit tests for utils methods and s3_uploader * Removed s3_bucket and s3_prefix click options * chore: Increase awareness of same file warning during package (aws#2946) * chore: increase awareness of same file warning during package * fix formatting & grammar Co-authored-by: Mathieu Grandis <73313235+mgrandis@users.noreply.github.com> * fix: Allow the base64Encoded field in REST Api, skip validation of unknown fields and validate missing statusCode for Http Api (aws#2941) * fix API Gateway emulator: - skip validating the non allowed fields for Http Api Gateway, as it always skip the unknown fields - add base64Encoded as an allowed field for Rest Api gateway - base64 decoding will be always done for Http API gateway if the lambda response isBase64Encoded is true regardless the content-type - validate if statusCode is missing in case of Http API, and payload version 1.0 * - accept "true", "True", "false", "False" as valid isBase64Encoded values. - Validate on other isBase64Encoded Values - add more integration && unit test cases * fix lint && black issues * use smaller image to test Base64 response * Fixed lint errors and added few unit tests * Make black happy * Added methods for deleting template artifacts * Wait method added for delete cf api * fix: pass copy of environment variables for keeping cache valid (aws#2943) * fix: pass copy of environment variables for keeping cache valid * add integ tests * update docs * make black happy Co-authored-by: Qingchuan Ma <69653965+qingchm@users.noreply.github.com> * Added LOG statements * Added and updated changes based on CR * Fixed the unit tests in artifact_exporter.py * Update HELP_TEXT in delete/command.py Co-authored-by: Chris Rehn <crehn@outlook.com> * fix: Skip build of Docker image if ImageUri is a valid ECR URL (aws#2934) (aws#2935) * Updated code based on Chris' comments * Added condition for resources that have deletionpolicy specified * Small changes and fixes based on the comments * Add condition to managed bucket policy (aws#2999) * Removed region prompt * Update appveyor.yml to do docker login on both dockerhub and Public ECR (aws#3005) (aws#3006) Co-authored-by: Wing Fung Lau <4760060+hawflau@users.noreply.github.com> * chore: bump version to 1.25.0 (aws#3007) Co-authored-by: Sriram Madapusi Vasudevan <3770774+sriram-mv@users.noreply.github.com> * temp: reduce python testing matrix (aws#3008) * temp: disable testing against python 3.8, and enabled 3.7 (aws#3009) * temp: disable testing against python 3.8, and enabled 3.7 * temp: disable testing against python 3.8, and enabled 3.7 & 3.6 * fix: enable all runtimes in python testing matrix (aws#3011) * revert: enable all runtimes in python testing matrix * fix indentation for yml * Added unit tests for ecr delete method and typing for methods * Reformatted delete_context and added option to skip user prompts * Removed return type from artifact_exporter for delete method * Added unit tests for artifact_exporter and delete_context * Added more unit tests for delete_context and artifact_exporter * chore: update to aws-sam-translator 1.37.0 (aws#3019) * chore: bump version to 1.26.0 (aws#3020) * Added more unit tests for delete_context and artifact_exporter * Added docs and comments for artifact_exporter and ecr_uploader * Added log statements in delete_context and some updates in unit tests * Changed force to no-prompts and updated ecr delete method error handling * chore: Improved --resolve-s3 option documentation and deployment without s3 error messages (aws#2983) * Improve documentation on --resolve-s3 option and improve s3 failure messages * Changed indentation for integration test on s3 error message * Fixed a typo in description * Improve spacing on help text for resolve-s3 option * Created a separate function for parsing ecr url in ecr_uploader * Reformatted Template class init to pass template_str and init template_dict * Changed how s3 url is obtained for resource_zip edge-case: aws:glue:job * Fixed edge case where resource artifact points to a path style url * run Make black * Made the parse s3 url funcs protected and defined a parent method and modified delete method for ResourceImageDict * Changed parse_ecr_url function name to parse_image_url * Defined UI for auto ecr deleton and method calls from companion_stack_manager * Added code for deleting repos from companion stack * Handle json templates deployed to cf * Changed the order of companion stack and ecr repos deletion * Handle delete_failed status for ecr companion stack and changed delete_stack to include retain_resources * Reformatted auto ecr deletion to handle deleting companion stack as input stack name * Fixed and added more unit tests for delete_context * When region is not provided, prompt user to enter profile and region * Removed region prompt and reading it from current session or assign a default instead * Added ECR resource in packageable_resources and refactored ecr companion stack deletion * Added log statements and unit tests for ECRResource * Better error handling for ecr delete_artifact * Revert "Merge remote-tracking branch 'wiltons-repo/feat/auto-ecr' into auto-ecr-delete" This reverts commit 0e159c2fa3630b874f13f19336802f6085a92de9, reversing changes made to 1675b7ed231b6472d38eeeeb25e39f6310bbb86f. * Added unit test for delete ecr repository * Fixed small string nits and added docstring for ECRResource * Added some unit tests for s3_uploader, ecr_uploader and delete_context * Updated to context refresh only when region and profile have non None values and removed unused class variable in delete_context * Added unit test for ResourceImageDict class methods Co-authored-by: Wilton Wang <CoshUS@users.noreply.github.com> Co-authored-by: Chris Rehn <crehn@outlook.com> Co-authored-by: Mathieu Grandis <73313235+mgrandis@users.noreply.github.com> Co-authored-by: Mehmet Nuri Deveci <5735811+mndeveci@users.noreply.github.com> Co-authored-by: Mohamed Elasmar <71043312+moelasmar@users.noreply.github.com> Co-authored-by: Qingchuan Ma <69653965+qingchm@users.noreply.github.com> Co-authored-by: Alexis Facques <mail@alexisfacques.com> Co-authored-by: Wing Fung Lau <4760060+hawflau@users.noreply.github.com> Co-authored-by: Sriram Madapusi Vasudevan <3770774+sriram-mv@users.noreply.github.com> * Sam delete bug fixes (aws#3122) * Fixed small bugs for no_prompts and for deleting artifacts * Updated integration tests and added test for termination protection for sam delete * Added few more integration tests for guided and non-guided delete * Updated handling termination protection and changed the order of deleting artifacts and stack * Added comments for delete artifacts to handle intrinsic ref functions and handled this for image resources * Added integration test for retaining s3 artifact * Changed option_name to the correct values in delete_context * Small UX fix and updated delete_prefix_artifacts method prefix to fetch S3 files * Added a note in s3_uploader.py about using the api list_objects_v2 * add sam delete to pyinstaller hooks file * fix typo * resolve pr comments Co-authored-by: hnnasit <84355507+hnnasit@users.noreply.github.com> Co-authored-by: Chris Rehn <crehn@outlook.com> Co-authored-by: Mehmet Nuri Deveci <5735811+mndeveci@users.noreply.github.com> Co-authored-by: Mathieu Grandis <73313235+mgrandis@users.noreply.github.com> Co-authored-by: Wilton Wang <CoshUS@users.noreply.github.com> Co-authored-by: Qingchuan Ma <69653965+qingchm@users.noreply.github.com> Co-authored-by: Alexis Facques <mail@alexisfacques.com> Co-authored-by: Wing Fung Lau <4760060+hawflau@users.noreply.github.com> Co-authored-by: Sriram Madapusi Vasudevan <3770774+sriram-mv@users.noreply.github.com>

* feat: Delete methods for CF stacks and S3 files (aws#2981) * Added methods for cf and s3 files and init UI * Added unit tests for utils methods and s3_uploader * Removed s3_bucket and s3_prefix click options * Fixed lint errors and added few unit tests * Make black happy * Added LOG statements * Added and updated changes based on CR * Fixed the unit tests in artifact_exporter.py * Update HELP_TEXT in delete/command.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Updated code based on Chris' comments * Small changes and fixes based on the comments * Removed region prompt * Update SAM context values for profile and region in delete_context.py * Added typing for get_cf_template_name method * Added stack_name prompt if the stack_name is not present in samconfig file * Replace [] with get() for stack-name in delete_context.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Delete template artifacts (aws#3022) * Added methods for cf and s3 files and init UI * Added unit tests for utils methods and s3_uploader * Removed s3_bucket and s3_prefix click options * chore: Increase awareness of same file warning during package (aws#2946) * chore: increase awareness of same file warning during package * fix formatting & grammar Co-authored-by: Mathieu Grandis <73313235+mgrandis@users.noreply.github.com> * fix: Allow the base64Encoded field in REST Api, skip validation of unknown fields and validate missing statusCode for Http Api (aws#2941) * fix API Gateway emulator: - skip validating the non allowed fields for Http Api Gateway, as it always skip the unknown fields - add base64Encoded as an allowed field for Rest Api gateway - base64 decoding will be always done for Http API gateway if the lambda response isBase64Encoded is true regardless the content-type - validate if statusCode is missing in case of Http API, and payload version 1.0 * - accept "true", "True", "false", "False" as valid isBase64Encoded values. - Validate on other isBase64Encoded Values - add more integration && unit test cases * fix lint && black issues * use smaller image to test Base64 response * Fixed lint errors and added few unit tests * Make black happy * Added methods for deleting template artifacts * Wait method added for delete cf api * Added LOG statements * Added and updated changes based on CR * Fixed the unit tests in artifact_exporter.py * Update HELP_TEXT in delete/command.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Updated code based on Chris' comments * Added condition for resources that have deletionpolicy specified * Small changes and fixes based on the comments * Removed region prompt * Added unit tests for ecr delete method and typing for methods * Reformatted delete_context and added option to skip user prompts * Removed return type from artifact_exporter for delete method * Added unit tests for artifact_exporter and delete_context * Added more unit tests for delete_context and artifact_exporter * Added more unit tests for delete_context and artifact_exporter * Added docs and comments for artifact_exporter and ecr_uploader * Added log statements in delete_context and some updates in unit tests * Changed force to no-prompts and updated ecr delete method error handling * Created a separate function for parsing ecr url in ecr_uploader * Reformatted Template class init to pass template_str and init template_dict * Changed how s3 url is obtained for resource_zip edge-case: aws:glue:job * Fixed edge case where resource artifact points to a path style url * run Make black * Made the parse s3 url funcs protected and defined a parent method and modified delete method for ResourceImageDict * Changed parse_ecr_url function name to parse_image_url Co-authored-by: Mehmet Nuri Deveci <5735811+mndeveci@users.noreply.github.com> Co-authored-by: Mathieu Grandis <73313235+mgrandis@users.noreply.github.com> Co-authored-by: Mohamed Elasmar <71043312+moelasmar@users.noreply.github.com> Co-authored-by: Chris Rehn <crehn@outlook.com> * Get s3 info cf template (aws#3050) * Added methods for cf and s3 files and init UI * Added unit tests for utils methods and s3_uploader * Removed s3_bucket and s3_prefix click options * chore: Increase awareness of same file warning during package (aws#2946) * chore: increase awareness of same file warning during package * fix formatting & grammar Co-authored-by: Mathieu Grandis <73313235+mgrandis@users.noreply.github.com> * fix: Allow the base64Encoded field in REST Api, skip validation of unknown fields and validate missing statusCode for Http Api (aws#2941) * fix API Gateway emulator: - skip validating the non allowed fields for Http Api Gateway, as it always skip the unknown fields - add base64Encoded as an allowed field for Rest Api gateway - base64 decoding will be always done for Http API gateway if the lambda response isBase64Encoded is true regardless the content-type - validate if statusCode is missing in case of Http API, and payload version 1.0 * - accept "true", "True", "false", "False" as valid isBase64Encoded values. - Validate on other isBase64Encoded Values - add more integration && unit test cases * fix lint && black issues * use smaller image to test Base64 response * Fixed lint errors and added few unit tests * Make black happy * Added methods for deleting template artifacts * Wait method added for delete cf api * Added LOG statements * Added and updated changes based on CR * Fixed the unit tests in artifact_exporter.py * Update HELP_TEXT in delete/command.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Updated code based on Chris' comments * Added condition for resources that have deletionpolicy specified * Small changes and fixes based on the comments * Removed region prompt * Added unit tests for ecr delete method and typing for methods * Reformatted delete_context and added option to skip user prompts * Removed return type from artifact_exporter for delete method * Added unit tests for artifact_exporter and delete_context * Added more unit tests for delete_context and artifact_exporter * Added more unit tests for delete_context and artifact_exporter * Added docs and comments for artifact_exporter and ecr_uploader * Added log statements in delete_context and some updates in unit tests * Changed force to no-prompts and updated ecr delete method error handling * Created a separate function for parsing ecr url in ecr_uploader * Reformatted Template class init to pass template_str and init template_dict * Changed how s3 url is obtained for resource_zip edge-case: aws:glue:job * Fixed edge case where resource artifact points to a path style url * run Make black * Made the parse s3 url funcs protected and defined a parent method and modified delete method for ResourceImageDict * Added methods to extract s3 info from cf template * Added testing for get_s3_info method for artifact_exporter and s3_uploader methods * Removed commented code and updated method docstring * Better error handling for s3 delete artifacts and fixed bug for getting s3 resources information * Changed get_s3_info to get_property_value and changed output text for s3 delete method Co-authored-by: Mehmet Nuri Deveci <5735811+mndeveci@users.noreply.github.com> Co-authored-by: Mathieu Grandis <73313235+mgrandis@users.noreply.github.com> Co-authored-by: Mohamed Elasmar <71043312+moelasmar@users.noreply.github.com> Co-authored-by: Chris Rehn <crehn@outlook.com> * Sam delete integration testing (aws#3076) * Added methods for cf and s3 files and init UI * Added unit tests for utils methods and s3_uploader * Removed s3_bucket and s3_prefix click options * chore: Increase awareness of same file warning during package (aws#2946) * chore: increase awareness of same file warning during package * fix formatting & grammar Co-authored-by: Mathieu Grandis <73313235+mgrandis@users.noreply.github.com> * fix: Allow the base64Encoded field in REST Api, skip validation of unknown fields and validate missing statusCode for Http Api (aws#2941) * fix API Gateway emulator: - skip validating the non allowed fields for Http Api Gateway, as it always skip the unknown fields - add base64Encoded as an allowed field for Rest Api gateway - base64 decoding will be always done for Http API gateway if the lambda response isBase64Encoded is true regardless the content-type - validate if statusCode is missing in case of Http API, and payload version 1.0 * - accept "true", "True", "false", "False" as valid isBase64Encoded values. - Validate on other isBase64Encoded Values - add more integration && unit test cases * fix lint && black issues * use smaller image to test Base64 response * Fixed lint errors and added few unit tests * Make black happy * Added methods for deleting template artifacts * Wait method added for delete cf api * Added LOG statements * Added and updated changes based on CR * Fixed the unit tests in artifact_exporter.py * Update HELP_TEXT in delete/command.py Co-authored-by: Chris Rehn <crehn@outlook.com> * Updated code based on Chris' comments * Added condition for resources that have deletionpolicy specified * Small changes and fixes based on the comments * Removed region prompt * Added unit tests for ecr delete method and typing for methods * Reformatted delete_context and added option to skip user prompts * Removed return type from artifact_exporter for delete method * Added unit tests for artifact_exporter and delete_context * Added more unit tests for delete_context and artifact_exporter * Added more unit tests for delete_context and artifact_exporter * Added docs and comments for artifact_exporter and ecr_uploader * Added log statements in delete_context and some updates in unit tests * Init integration tests methods for sam delete * Added more template files as a list for sam delete integration testing * Changed force to no-prompts and updated ecr delete method error handling * Created a separate function for parsing ecr url in ecr_uploader * Reformatted Template class init to pass template_str and init template_dict * Changed how s3 url is obtained for resource_zip edge-case: aws:glue:job * Fixed edge case where resource artifact points to a path style url * run Make black * Added 2 more integrations tests no_stack_deployed and delete for image type resources * Made the parse s3 url funcs protected and defined a parent method and modified delete method for ResourceImageDict * Added methods to extract s3 info from cf template * Added testing for get_s3_info method for artifact_exporter and s3_uploader methods * Added few more integration tests for delete * Merged delete-template-artifacts branch code and updated this branch code * Added tests for no prefix present for zip and image * Added a check to confirm input stack is deleted for all the integration tests * Removed commented code and updated method docstring * Better error handling for s3 delete artifacts and fixed bug for getting s3 resources information * Changed get_s3_info to get_property_value and changed output text for s3 delete method * Added integration test for deleting nested stacks Co-authored-by: Mehmet Nuri Deveci <5735811+mndeveci@users.noreply.github.com> Co-authored-by: Mathieu Grandis <73313235+mgrandis@users.noreply.github.com> Co-authored-by: Mohamed Elasmar <71043312+moelasmar@users.noreply.github.com> Co-authored-by: Chris Rehn <crehn@outlook.com> * Auto ECR Companion Stack Deletion (aws#3080) * Added ecr_bootstrap * Added companion_stack_manager * Added Companion Stack Manager * Added update_companion_stack * Updated companion_stack_builder File Name * Formatted with Black * Updated get_unreferenced_repos * Updated guided_context to Use Companion Stack * Added Delete Auto Create ECR Repo Prompt * Updated prompt_image_repository Flow * Added --resolve-image-repos * Addressed Some of Pylint Issues * Updated Helper Text * Updated Comments * Fixed Typing * Removed Unused Imports * Updated Unit Tests * Updated UX and Fixed Windows ANSI * Updated Unit Tests * Fixed Import Order * Added Ignore Import Check * Added Integration Tests * Updated help text. Co-authored-by: Chris Rehn <crehn@outlook.com> * Added Comments for Name Generation * Updated Image Option Validator * Updated CompanionStackBuilder to Use Dict instead of String * Fixed Argument Ordering * Added Mapping Information to Help Text * Updated delete_unreferenced_repos Doc String * Updated sync_repos Doc String * Added Justification for ECR Repo Physical ID * Refactored to be Less Coupled * Refactored for prompt_specify_repos * Fixed Unit Test * Moved WaiterConfig Out of Methods * Updated Typing * Updated Managed S3 Template to be Dict * Fixed Typo * Added Comments for _save_image_repositories * Fixed Pylint Issue * Added Missing Check for unreferenced_repo_uris * Updated Variable Name * Fixed Typo * Updated Windows Check to Use platform.system() * Updated update_companion_stack Logic * Fixed Comment Typo * Fixed Typos * Fixed Test Name * Added methods for cf and s3 files and init UI * Added unit tests for utils methods and s3_uploader * Removed s3_bucket and s3_prefix click options * chore: Increase awareness of same file warning during package (aws#2946) * chore: increase awareness of same file warning during package * fix formatting & grammar Co-authored-by: Mathieu Grandis <73313235+mgrandis@users.noreply.github.com> * fix: Allow the base64Encoded field in REST Api, skip validation of unknown fields and validate missing statusCode for Http Api (aws#2941) * fix API Gateway emulator: - skip validating the non allowed fields for Http Api Gateway, as it always skip the unknown fields - add base64Encoded as an allowed field for Rest Api gateway - base64 decoding will be always done for Http API gateway if the lambda response isBase64Encoded is true regardless the content-type - validate if statusCode is missing in case of Http API, and payload version 1.0 * - accept "true", "True", "false", "False" as valid isBase64Encoded values. - Validate on other isBase64Encoded Values - add more integration && unit test cases * fix lint && black issues * use smaller image to test Base64 response * Fixed lint errors and added few unit tests * Make black happy * Added methods for deleting template artifacts * Wait method added for delete cf api * fix: pass copy of environment variables for keeping cache valid (aws#2943) * fix: pass copy of environment variables for keeping cache valid * add integ tests * update docs * make black happy Co-authored-by: Qingchuan Ma <69653965+qingchm@users.noreply.github.com> * Added LOG statements * Added and updated changes based on CR * Fixed the unit tests in artifact_exporter.py * Update HELP_TEXT in delete/command.py Co-authored-by: Chris Rehn <crehn@outlook.com> * fix: Skip build of Docker image if ImageUri is a valid ECR URL (aws#2934) (aws#2935) * Updated code based on Chris' comments * Added condition for resources that have deletionpolicy specified * Small changes and fixes based on the comments * Add condition to managed bucket policy (aws#2999) * Removed region prompt * Update appveyor.yml to do docker login on both dockerhub and Public ECR (aws#3005) (aws#3006) Co-authored-by: Wing Fung Lau <4760060+hawflau@users.noreply.github.com> * chore: bump version to 1.25.0 (aws#3007) Co-authored-by: Sriram Madapusi Vasudevan <3770774+sriram-mv@users.noreply.github.com> * temp: reduce python testing matrix (aws#3008) * temp: disable testing against python 3.8, and enabled 3.7 (aws#3009) * temp: disable testing against python 3.8, and enabled 3.7 * temp: disable testing against python 3.8, and enabled 3.7 & 3.6 * fix: enable all runtimes in python testing matrix (aws#3011) * revert: enable all runtimes in python testing matrix * fix indentation for yml * Added unit tests for ecr delete method and typing for methods * Reformatted delete_context and added option to skip user prompts * Removed return type from artifact_exporter for delete method * Added unit tests for artifact_exporter and delete_context * Added more unit tests for delete_context and artifact_exporter * chore: update to aws-sam-translator 1.37.0 (aws#3019) * chore: bump version to 1.26.0 (aws#3020) * Added more unit tests for delete_context and artifact_exporter * Added docs and comments for artifact_exporter and ecr_uploader * Added log statements in delete_context and some updates in unit tests * Changed force to no-prompts and updated ecr delete method error handling * chore: Improved --resolve-s3 option documentation and deployment without s3 error messages (aws#2983) * Improve documentation on --resolve-s3 option and improve s3 failure messages * Changed indentation for integration test on s3 error message * Fixed a typo in description * Improve spacing on help text for resolve-s3 option * Created a separate function for parsing ecr url in ecr_uploader * Reformatted Template class init to pass template_str and init template_dict * Changed how s3 url is obtained for resource_zip edge-case: aws:glue:job * Fixed edge case where resource artifact points to a path style url * run Make black * Made the parse s3 url funcs protected and defined a parent method and modified delete method for ResourceImageDict * Changed parse_ecr_url function name to parse_image_url * Defined UI for auto ecr deleton and method calls from companion_stack_manager * Added code for deleting repos from companion stack * Handle json templates deployed to cf * Changed the order of companion stack and ecr repos deletion * Handle delete_failed status for ecr companion stack and changed delete_stack to include retain_resources * Reformatted auto ecr deletion to handle deleting companion stack as input stack name * Fixed and added more unit tests for delete_context * When region is not provided, prompt user to enter profile and region * Removed region prompt and reading it from current session or assign a default instead * Added ECR resource in packageable_resources and refactored ecr companion stack deletion * Added log statements and unit tests for ECRResource * Better error handling for ecr delete_artifact * Revert "Merge remote-tracking branch 'wiltons-repo/feat/auto-ecr' into auto-ecr-delete" This reverts commit 0e159c2fa3630b874f13f19336802f6085a92de9, reversing changes made to 1675b7ed231b6472d38eeeeb25e39f6310bbb86f. * Added unit test for delete ecr repository * Fixed small string nits and added docstring for ECRResource * Added some unit tests for s3_uploader, ecr_uploader and delete_context * Updated to context refresh only when region and profile have non None values and removed unused class variable in delete_context * Added unit test for ResourceImageDict class methods Co-authored-by: Wilton Wang <CoshUS@users.noreply.github.com> Co-authored-by: Chris Rehn <crehn@outlook.com> Co-authored-by: Mathieu Grandis <73313235+mgrandis@users.noreply.github.com> Co-authored-by: Mehmet Nuri Deveci <5735811+mndeveci@users.noreply.github.com> Co-authored-by: Mohamed Elasmar <71043312+moelasmar@users.noreply.github.com> Co-authored-by: Qingchuan Ma <69653965+qingchm@users.noreply.github.com> Co-authored-by: Alexis Facques <mail@alexisfacques.com> Co-authored-by: Wing Fung Lau <4760060+hawflau@users.noreply.github.com> Co-authored-by: Sriram Madapusi Vasudevan <3770774+sriram-mv@users.noreply.github.com> * Sam delete bug fixes (aws#3122) * Fixed small bugs for no_prompts and for deleting artifacts * Updated integration tests and added test for termination protection for sam delete * Added few more integration tests for guided and non-guided delete * Updated handling termination protection and changed the order of deleting artifacts and stack * Added comments for delete artifacts to handle intrinsic ref functions and handled this for image resources * Added integration test for retaining s3 artifact * Changed option_name to the correct values in delete_context * Small UX fix and updated delete_prefix_artifacts method prefix to fetch S3 files * Added a note in s3_uploader.py about using the api list_objects_v2 * add sam delete to pyinstaller hooks file * fix typo * resolve pr comments * fix refactoring issue, and add unit testing Co-authored-by: hnnasit <84355507+hnnasit@users.noreply.github.com> Co-authored-by: Chris Rehn <crehn@outlook.com> Co-authored-by: Mehmet Nuri Deveci <5735811+mndeveci@users.noreply.github.com> Co-authored-by: Mathieu Grandis <73313235+mgrandis@users.noreply.github.com> Co-authored-by: Wilton Wang <CoshUS@users.noreply.github.com> Co-authored-by: Qingchuan Ma <69653965+qingchm@users.noreply.github.com> Co-authored-by: Alexis Facques <mail@alexisfacques.com> Co-authored-by: Wing Fung Lau <4760060+hawflau@users.noreply.github.com> Co-authored-by: Sriram Madapusi Vasudevan <3770774+sriram-mv@users.noreply.github.com>

joguSD added the closing-soon This issue will automatically close in 4 days unless further comments are made. label Nov 30, 2017

jamesls closed this as completed Dec 11, 2017

delgod added a commit to Percona-Lab/jenkins-pipelines that referenced this issue May 23, 2018

add workaround for aws/aws-cli#3007

79b0b74

delgod added a commit to Percona-Lab/jenkins-pipelines that referenced this issue May 23, 2018

add workaround for aws/aws-cli#3007

7efac7e

thoward-godaddy pushed a commit to thoward-godaddy/aws-cli that referenced this issue Feb 12, 2022

chore: bump version to 1.25.0 (aws#3007)

f1e0569

Co-authored-by: Sriram Madapusi Vasudevan <3770774+sriram-mv@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

s3 sync broken in 1.14.0 - unexpected keyword argument 'max_bandwidth' #3007

s3 sync broken in 1.14.0 - unexpected keyword argument 'max_bandwidth' #3007

abrom commented Nov 30, 2017 •

edited

Loading

hafizramly commented Nov 30, 2017

joguSD commented Nov 30, 2017

kelleydon commented Nov 30, 2017

joguSD commented Nov 30, 2017

hafizramly commented Dec 4, 2017

rflaperuta commented Dec 5, 2017

jamesls commented Dec 11, 2017

jgramoll commented Dec 29, 2017 •

edited

Loading

sakers commented Jan 10, 2018 •

edited

Loading

meowtochondria commented Apr 13, 2018 •

edited

Loading

karnauskas commented May 11, 2018

CiliaFred commented May 11, 2018 •

edited

Loading

pbudzon commented May 20, 2018 •

edited

Loading

heraldb commented May 31, 2018

IlariExove commented Jun 14, 2018

s3 sync broken in 1.14.0 - unexpected keyword argument 'max_bandwidth' #3007

s3 sync broken in 1.14.0 - unexpected keyword argument 'max_bandwidth' #3007

Comments

abrom commented Nov 30, 2017 • edited Loading

hafizramly commented Nov 30, 2017

joguSD commented Nov 30, 2017

kelleydon commented Nov 30, 2017

joguSD commented Nov 30, 2017

hafizramly commented Dec 4, 2017

rflaperuta commented Dec 5, 2017

jamesls commented Dec 11, 2017

jgramoll commented Dec 29, 2017 • edited Loading

sakers commented Jan 10, 2018 • edited Loading

meowtochondria commented Apr 13, 2018 • edited Loading

karnauskas commented May 11, 2018

CiliaFred commented May 11, 2018 • edited Loading

pbudzon commented May 20, 2018 • edited Loading

heraldb commented May 31, 2018

IlariExove commented Jun 14, 2018

abrom commented Nov 30, 2017 •

edited

Loading

jgramoll commented Dec 29, 2017 •

edited

Loading

sakers commented Jan 10, 2018 •

edited

Loading

meowtochondria commented Apr 13, 2018 •

edited

Loading

CiliaFred commented May 11, 2018 •

edited

Loading

pbudzon commented May 20, 2018 •

edited

Loading