s3cmd v2.0.0 with Python 2.7.9 using SSL behind Squid Proxy, getting "ERROR: S3 error: 400 (Bad Request)" #905

dannyk81 · 2017-07-28T01:14:47Z

Hey guys,

I'm running s3cmd v2.0.0 on Debian Jessie (8.7) with Python 2.7.9.

We have a Squid proxy in place and when setting up s3cmd with SSL (use_https = True) s3cmd fails with ERROR: S3 error: 400 (Bad Request) error. With use_https = False everything works perfectly.

UPDATE: I performed two tests
UPDATE 2: test 3 - 5

I installed aws-cli (v1.11.127) on the same node and it works just fine (I verified it uses our proxy and HTTPS):

$ aws s3 ls
2017-07-20 21:04:20 *****-backups-***
2017-07-26 01:22:42 *****-backups-***

I configured the server to not use a Proxy and it works too
Tried with Python 3 (3.4.2) -> doesn't work
Tried with Buckets in different regions (Ireland, Oregen) -> doesn't work
Upgraded Squid from 3.4.8 to 3.5.25 -> didn't help

Based on above, it seems like some kind of combination between s3cmd and Squid that just doesn't work. However, we have many other HTTPS connections going through this Proxy using the CONNECT method without any issues...

Here is a debug snippet:

$ s3cmd ls --debug
DEBUG: s3cmd version 2.0.0
DEBUG: ConfigParser: Reading file '/home/backup/.s3cfg'
DEBUG: ConfigParser: access_key->AK...17_chars...A
DEBUG: ConfigParser: access_token->
DEBUG: ConfigParser: add_encoding_exts->
DEBUG: ConfigParser: add_headers->
DEBUG: ConfigParser: bucket_location->US
DEBUG: ConfigParser: ca_certs_file->
DEBUG: ConfigParser: cache_file->
DEBUG: ConfigParser: check_ssl_certificate->True
DEBUG: ConfigParser: check_ssl_hostname->True
DEBUG: ConfigParser: cloudfront_host->cloudfront.amazonaws.com
DEBUG: ConfigParser: default_mime_type->binary/octet-stream
DEBUG: ConfigParser: delay_updates->False
DEBUG: ConfigParser: delete_after->False
DEBUG: ConfigParser: delete_after_fetch->False
DEBUG: ConfigParser: delete_removed->False
DEBUG: ConfigParser: dry_run->False
DEBUG: ConfigParser: enable_multipart->True
DEBUG: ConfigParser: encoding->UTF-8
DEBUG: ConfigParser: encrypt->False
DEBUG: ConfigParser: expiry_date->
DEBUG: ConfigParser: expiry_days->
DEBUG: ConfigParser: expiry_prefix->
DEBUG: ConfigParser: follow_symlinks->False
DEBUG: ConfigParser: force->False
DEBUG: ConfigParser: get_continue->False
DEBUG: ConfigParser: gpg_command->/usr/bin/gpg
DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_passphrase->f0...3_chars...R
DEBUG: ConfigParser: guess_mime_type->True
DEBUG: ConfigParser: host_base->s3.amazonaws.com
DEBUG: ConfigParser: host_bucket->%(bucket)s.s3.amazonaws.com
DEBUG: ConfigParser: human_readable_sizes->False
DEBUG: ConfigParser: invalidate_default_index_on_cf->False
DEBUG: ConfigParser: invalidate_default_index_root_on_cf->True
DEBUG: ConfigParser: invalidate_on_cf->False
DEBUG: ConfigParser: kms_key->
DEBUG: ConfigParser: limit->-1
DEBUG: ConfigParser: limitrate->0
DEBUG: ConfigParser: list_md5->False
DEBUG: ConfigParser: log_target_prefix->
DEBUG: ConfigParser: long_listing->False
DEBUG: ConfigParser: max_delete->-1
DEBUG: ConfigParser: mime_type->
DEBUG: ConfigParser: multipart_chunk_size_mb->15
DEBUG: ConfigParser: multipart_max_chunks->10000
DEBUG: ConfigParser: preserve_attrs->True
DEBUG: ConfigParser: progress_meter->True
DEBUG: ConfigParser: proxy_host->fpx.prd.mia.novumproject.com
DEBUG: ConfigParser: proxy_port->8080
DEBUG: ConfigParser: put_continue->False
DEBUG: ConfigParser: recursive->False
DEBUG: ConfigParser: recv_chunk->65536
DEBUG: ConfigParser: reduced_redundancy->False
DEBUG: ConfigParser: requester_pays->False
DEBUG: ConfigParser: restore_days->1
DEBUG: ConfigParser: restore_priority->Standard
DEBUG: ConfigParser: secret_key->9U...37_chars...A
DEBUG: ConfigParser: send_chunk->65536
DEBUG: ConfigParser: server_side_encryption->False
DEBUG: ConfigParser: signature_v2->False
DEBUG: ConfigParser: simpledb_host->sdb.amazonaws.com
DEBUG: ConfigParser: skip_existing->False
DEBUG: ConfigParser: socket_timeout->300
DEBUG: ConfigParser: stats->False
DEBUG: ConfigParser: stop_on_error->False
DEBUG: ConfigParser: storage_class->
DEBUG: ConfigParser: urlencoding_mode->normal
DEBUG: ConfigParser: use_http_expect->False
DEBUG: ConfigParser: use_https->True
DEBUG: ConfigParser: use_mime_magic->True
DEBUG: ConfigParser: verbosity->WARNING
DEBUG: ConfigParser: website_endpoint->http://%(bucket)s.s3-website-%(location)s.amazonaws.com/
DEBUG: ConfigParser: website_error->
DEBUG: ConfigParser: website_index->index.html
DEBUG: Updating Config.Config cache_file ->
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'ls' using UTF-8
DEBUG: Command: ls
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v2
DEBUG: SignHeaders: u'GET\n\n\n\nx-amz-date:Fri, 28 Jul 2017 01:12:54 +0000\n/'
DEBUG: Processing request, please wait...
DEBUG: get_hostname(None): s3.amazonaws.com
DEBUG: ConnMan.get(): creating new connection: proxy://fpx.prd.mia.novumproject.com:8080
DEBUG: Using ca_certs_file None
DEBUG: httplib.HTTPSConnection() has only context
DEBUG: proxied HTTPSConnection(fpx.prd.mia.novumproject.com, 8080)
DEBUG: tunnel to s3.amazonaws.com, None
DEBUG: format_uri(): /
DEBUG: Sending request method_string='GET', uri=u'/', headers={'Authorization': u'AWS AKIAJC6JE7KDKR2N4WKA:7e0JA1RN3bcDPMUlFyekxjUcllk=', 'x-amz-date': 'Fri, 28 Jul 2017 01:12:54 +0000'}, body=(0 bytes)
DEBUG: ConnMan.put(): closing proxy connection (keep-alive not yet supported)
DEBUG: Response:
{'data': '',
 'headers': {'connection': 'close',
             'date': 'Fri, 28 Jul 2017 01:12:53 GMT',
             'server': 'AmazonS3',
             'transfer-encoding': 'chunked'},
 'reason': 'Bad Request',
 'status': 400}
DEBUG: S3Error: 400 (Bad Request)
DEBUG: HttpHeader: transfer-encoding: chunked
DEBUG: HttpHeader: date: Fri, 28 Jul 2017 01:12:53 GMT
DEBUG: HttpHeader: connection: close
DEBUG: HttpHeader: server: AmazonS3
ERROR: S3 error: 400 (Bad Request)

This seems like an issue with our Proxy, however I can't seem to figure out what.

The text was updated successfully, but these errors were encountered:

dannyk81 · 2017-08-01T15:36:52Z

After several tries and tests, I figured I would try an older version and downgraded to v1.6.1 - to my surprise, it works!

Seems like there's regression from v1.6.1 -> v2.0.0 with regard to proxied SSL connections.

here's a debug output using v1.6.1:

$ ./s3cmd ls --debug
DEBUG: s3cmd version 1.6.1
DEBUG: ConfigParser: Reading file '/root/.s3cfg'
DEBUG: ConfigParser: access_key->AK...17_chars...A
DEBUG: ConfigParser: access_token->
DEBUG: ConfigParser: add_encoding_exts->
DEBUG: ConfigParser: add_headers->
DEBUG: ConfigParser: bucket_location->US
DEBUG: ConfigParser: ca_certs_file->
DEBUG: ConfigParser: cache_file->
DEBUG: ConfigParser: check_ssl_certificate->True
DEBUG: ConfigParser: check_ssl_hostname->True
DEBUG: ConfigParser: cloudfront_host->cloudfront.amazonaws.com
DEBUG: ConfigParser: default_mime_type->binary/octet-stream
DEBUG: ConfigParser: delay_updates->False
DEBUG: ConfigParser: delete_after->False
DEBUG: ConfigParser: delete_after_fetch->False
DEBUG: ConfigParser: delete_removed->False
DEBUG: ConfigParser: dry_run->False
DEBUG: ConfigParser: enable_multipart->True
DEBUG: ConfigParser: encoding->UTF-8
DEBUG: ConfigParser: encrypt->False
DEBUG: ConfigParser: expiry_date->
DEBUG: ConfigParser: expiry_days->
DEBUG: ConfigParser: expiry_prefix->
DEBUG: ConfigParser: follow_symlinks->False
DEBUG: ConfigParser: force->False
DEBUG: ConfigParser: get_continue->False
DEBUG: ConfigParser: gpg_command->/usr/bin/gpg
DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_passphrase->...-3_chars...
DEBUG: ConfigParser: guess_mime_type->True
DEBUG: ConfigParser: host_base->s3.amazonaws.com
DEBUG: ConfigParser: host_bucket->%(bucket)s.s3.amazonaws.com
DEBUG: ConfigParser: human_readable_sizes->False
DEBUG: ConfigParser: invalidate_default_index_on_cf->False
DEBUG: ConfigParser: invalidate_default_index_root_on_cf->True
DEBUG: ConfigParser: invalidate_on_cf->False
DEBUG: ConfigParser: kms_key->46e5d2e4-1037-4c76-b0da-fdf1253dc08e
DEBUG: ConfigParser: limit->-1
DEBUG: ConfigParser: limitrate->15m
DEBUG: ConfigParser: list_md5->False
DEBUG: ConfigParser: log_target_prefix->
DEBUG: ConfigParser: long_listing->False
DEBUG: ConfigParser: max_delete->-1
DEBUG: ConfigParser: mime_type->
DEBUG: ConfigParser: multipart_chunk_size_mb->15
DEBUG: ConfigParser: multipart_max_chunks->10000
DEBUG: ConfigParser: preserve_attrs->True
DEBUG: ConfigParser: progress_meter->False
DEBUG: ConfigParser: proxy_host->fpx.prd.mia.novumproject.com
DEBUG: ConfigParser: proxy_port->8080
DEBUG: ConfigParser: put_continue->False
DEBUG: ConfigParser: recursive->False
DEBUG: ConfigParser: recv_chunk->65536
DEBUG: ConfigParser: reduced_redundancy->False
DEBUG: ConfigParser: requester_pays->False
DEBUG: ConfigParser: restore_days->1
DEBUG: ConfigParser: restore_priority->Standard
DEBUG: ConfigParser: secret_key->9U...37_chars...A
DEBUG: ConfigParser: send_chunk->65536
DEBUG: ConfigParser: server_side_encryption->False
DEBUG: ConfigParser: signature_v2->False
DEBUG: ConfigParser: simpledb_host->sdb.amazonaws.com
DEBUG: ConfigParser: skip_existing->False
DEBUG: ConfigParser: socket_timeout->300
DEBUG: ConfigParser: stats->False
DEBUG: ConfigParser: stop_on_error->False
DEBUG: ConfigParser: storage_class->
DEBUG: ConfigParser: urlencoding_mode->normal
DEBUG: ConfigParser: use_http_expect->False
DEBUG: ConfigParser: use_https->True
DEBUG: ConfigParser: use_mime_magic->True
DEBUG: ConfigParser: verbosity->WARNING
DEBUG: ConfigParser: website_endpoint->http://%(bucket)s.s3-website-%(location)s.amazonaws.com/
DEBUG: ConfigParser: website_error->
DEBUG: ConfigParser: website_index->index.html
DEBUG: Updating Config.Config cache_file ->
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'ls' using UTF-8
DEBUG: Command: ls
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v2
DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Tue, 01 Aug 2017 15:28:15 +0000\n/'
DEBUG: Processing request, please wait...
DEBUG: get_hostname(None): s3.amazonaws.com
DEBUG: ConnMan.get(): creating new connection: proxy://fpx.prd.mia.novumproject.com:8080
DEBUG: Using ca_certs_file None
DEBUG: httplib.HTTPSConnection() has only context
DEBUG: proxied HTTPSConnection(fpx.prd.mia.novumproject.com, 3128)
DEBUG: tunnel to s3.amazonaws.com
DEBUG: get_hostname(None): s3.amazonaws.com
DEBUG: format_uri(): http://s3.amazonaws.com/
DEBUG: Sending request method_string='GET', uri='http://s3.amazonaws.com/', headers={'Authorization': 'AWS AKIAJC6JE7KDKR2N4WKA:aEp3QZhcOSk1NMO66HK/bIHN+qY=', 'x-amz-date': 'Tue, 01 Aug 2017 15:28:15 +0000'}, body=(0 bytes)
DEBUG: Response: {'status': 200, 'headers': {'x-amz-id-2': 'AlhLSSkgNYSuTlgbnPQg/cP9NFEut3E/KkOvSWGouPX1W0+eru/GBcgCWG8VxMj1r4v3cTtN6q8=', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'x-amz-request-id': '874A191EB16052BE', 'date': 'Tue, 01 Aug 2017 15:28:17 GMT', 'content-type': 'application/xml'}, 'reason': 'OK', 'data': '<?xml version="1.0" encoding="UTF-8"?>\n<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>08e79931a21c187af661a77556469a1e4f00df06101b1975c04f38926376daf2</ID><DisplayName>sre+awsnovum</DisplayName></Owner><Buckets><Bucket><Name>bucket-1</Name><CreationDate>2017-07-20T21:04:20.000Z</CreationDate></Bucket><Bucket><Name>bucket-2</Name><CreationDate>2017-07-26T01:22:42.000Z</CreationDate></Bucket><Bucket><Name>bucket-3</Name><CreationDate>2017-07-28T14:12:11.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>'}
DEBUG: ConnMan.put(): closing proxy connection (keep-alive not yet supported)
2017-07-20 21:04  s3://<bucket-1>
2017-07-26 01:22  s3://<bucket-2>
2017-07-28 14:12  s3://<bucket-3>

dannyk81 · 2017-08-01T23:00:16Z

Some further testing revealed that although with v1.6.1 the commands ls & put are working, the get command fails :-( this is really frustrating...

Debug output:

$ s3cmd get s3://<bucket>/test.txt --debug
DEBUG: s3cmd version 1.6.1
DEBUG: ConfigParser: Reading file '/home/backup/.s3cfg'
DEBUG: ConfigParser: access_key->AK...17_chars...A
DEBUG: ConfigParser: access_token->
DEBUG: ConfigParser: add_encoding_exts->
DEBUG: ConfigParser: add_headers->
DEBUG: ConfigParser: bucket_location->US
DEBUG: ConfigParser: ca_certs_file->
DEBUG: ConfigParser: cache_file->
DEBUG: ConfigParser: check_ssl_certificate->True
DEBUG: ConfigParser: check_ssl_hostname->True
DEBUG: ConfigParser: cloudfront_host->cloudfront.amazonaws.com
DEBUG: ConfigParser: default_mime_type->binary/octet-stream
DEBUG: ConfigParser: delay_updates->False
DEBUG: ConfigParser: delete_after->False
DEBUG: ConfigParser: delete_after_fetch->False
DEBUG: ConfigParser: delete_removed->False
DEBUG: ConfigParser: dry_run->False
DEBUG: ConfigParser: enable_multipart->True
DEBUG: ConfigParser: encoding->UTF-8
DEBUG: ConfigParser: encrypt->False
DEBUG: ConfigParser: expiry_date->
DEBUG: ConfigParser: expiry_days->
DEBUG: ConfigParser: expiry_prefix->
DEBUG: ConfigParser: follow_symlinks->False
DEBUG: ConfigParser: force->False
DEBUG: ConfigParser: get_continue->False
DEBUG: ConfigParser: gpg_command->/usr/bin/gpg
DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_passphrase->...-3_chars...
DEBUG: ConfigParser: guess_mime_type->True
DEBUG: ConfigParser: host_base->s3.amazonaws.com
DEBUG: ConfigParser: host_bucket->%(bucket)s.s3.amazonaws.com
DEBUG: ConfigParser: human_readable_sizes->False
DEBUG: ConfigParser: invalidate_default_index_on_cf->False
DEBUG: ConfigParser: invalidate_default_index_root_on_cf->True
DEBUG: ConfigParser: invalidate_on_cf->False
DEBUG: ConfigParser: kms_key->adeec8b5-35ef-4c1d-9251-64ee159b68e0
DEBUG: ConfigParser: limit->-1
DEBUG: ConfigParser: limitrate->30m
DEBUG: ConfigParser: list_md5->False
DEBUG: ConfigParser: log_target_prefix->
DEBUG: ConfigParser: long_listing->False
DEBUG: ConfigParser: max_delete->-1
DEBUG: ConfigParser: mime_type->
DEBUG: ConfigParser: multipart_chunk_size_mb->15
DEBUG: ConfigParser: multipart_max_chunks->10000
DEBUG: ConfigParser: preserve_attrs->True
DEBUG: ConfigParser: progress_meter->False
DEBUG: ConfigParser: proxy_host->fpx.prd.mia.novumproject.com
DEBUG: ConfigParser: proxy_port->8080
DEBUG: ConfigParser: put_continue->False
DEBUG: ConfigParser: recursive->False
DEBUG: ConfigParser: recv_chunk->65536
DEBUG: ConfigParser: reduced_redundancy->False
DEBUG: ConfigParser: requester_pays->False
DEBUG: ConfigParser: restore_days->1
DEBUG: ConfigParser: restore_priority->Standard
DEBUG: ConfigParser: secret_key->9U...37_chars...A
DEBUG: ConfigParser: send_chunk->65536
DEBUG: ConfigParser: server_side_encryption->False
DEBUG: ConfigParser: signature_v2->False
DEBUG: ConfigParser: simpledb_host->sdb.amazonaws.com
DEBUG: ConfigParser: skip_existing->False
DEBUG: ConfigParser: socket_timeout->300
DEBUG: ConfigParser: stats->False
DEBUG: ConfigParser: stop_on_error->False
DEBUG: ConfigParser: storage_class->
DEBUG: ConfigParser: urlencoding_mode->normal
DEBUG: ConfigParser: use_http_expect->False
DEBUG: ConfigParser: use_https->True
DEBUG: ConfigParser: use_mime_magic->True
DEBUG: ConfigParser: verbosity->WARNING
DEBUG: ConfigParser: website_endpoint->http://%(bucket)s.s3-website-%(location)s.amazonaws.com/
DEBUG: ConfigParser: website_error->
DEBUG: ConfigParser: website_index->index.html
DEBUG: Updating Config.Config cache_file ->
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'get' using UTF-8
DEBUG: Unicodising 's3://<bucket>/test.txt' using UTF-8
DEBUG: Command: get
DEBUG: DeUnicodising u'test.txt' using UTF-8
DEBUG: Unicodising 'test.txt' using UTF-8
DEBUG: Applying --exclude/--include
DEBUG: CHECK: test.txt
DEBUG: PASS: u'test.txt'
INFO: Summary: 1 remote files to download
DEBUG: DeUnicodising u'./test.txt' using UTF-8
DEBUG: Unicodising './test.txt' using UTF-8
DEBUG: DeUnicodising u'./test.txt' using UTF-8
DEBUG: DeUnicodising u'./test.txt' using UTF-8
DEBUG: String 'test.txt' encoded to 'test.txt'
DEBUG: CreateRequest: resource[uri]=/test.txt
DEBUG: Using signature v4
DEBUG: get_hostname(<bucket>): <bucket>.s3.amazonaws.com
DEBUG: canonical_headers = host:<bucket>.s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20170801T225835Z

DEBUG: Canonical Request:
GET
/test.txt

host:<bucket>.s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20170801T225835Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
----------------------
DEBUG: signature-v4 headers: {'x-amz-content-sha256': 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': 'AWS4-HMAC-SHA256 Credential=AKIAJC6JE7KDKR2N4WKA/20170801/US/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=478c8f266f5fc388d927e53bdc826eea012ecaf85fda7285fd9b1efc7b3084e2', 'x-amz-date': '20170801T225835Z'}
DEBUG: Unicodising './test.txt' using UTF-8
INFO: Receiving file './test.txt', please wait...
DEBUG: get_hostname(<bucket>): <bucket>.s3.amazonaws.com
DEBUG: ConnMan.get(): creating new connection: proxy://fpx.prd.mia.novumproject.com:8080
DEBUG: Using ca_certs_file None
DEBUG: httplib.HTTPSConnection() has only context
DEBUG: proxied HTTPSConnection(fpx.prd.mia.novumproject.com, 8080)
DEBUG: tunnel to <bucket>.s3.amazonaws.com
DEBUG: get_hostname(<bucket>): <bucket>.s3.amazonaws.com
DEBUG: format_uri(): http://<bucket>.s3.amazonaws.com/test.txt
DEBUG: Response: {'status': 400, 'headers': {'x-amz-id-2': 'ri2UdNfNjQBdiRr+t/eFErjkekjr03nzRTwUwQOoTZcPWzFJcLuVUQswfEuu3L3edkh8KbTotZ4=', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'connection': 'close', 'x-amz-request-id': 'E0A02CACAC726579', 'date': 'Tue, 01 Aug 2017 22:58:35 GMT', 'content-type': 'application/xml'}, 'reason': 'Bad Request'}
DEBUG: Falling back to signature v2
DEBUG: Using signature v2
DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:Tue, 01 Aug 2017 22:58:35 +0000\n/<bucket>/test.txt'
DEBUG: Unicodising './test.txt' using UTF-8
INFO: Receiving file './test.txt', please wait...
DEBUG: get_hostname(<bucket>): <bucket>.s3.amazonaws.com
DEBUG: ConnMan.get(): creating new connection: proxy://fpx.prd.mia.novumproject.com:8080
DEBUG: httplib.HTTPSConnection() has only context
DEBUG: proxied HTTPSConnection(fpx.prd.mia.novumproject.com, 8080)
DEBUG: tunnel to <bucket>.s3.amazonaws.com
DEBUG: get_hostname(<bucket>): <bucket>.s3.amazonaws.com
DEBUG: format_uri(): http://<bucket>.s3.amazonaws.com/test.txt
DEBUG: Response: {'status': 400, 'headers': {'x-amz-region': 'us-east-1', 'x-amz-id-2': '30gjCVw0WTi3ZAWRplPq9DuTyHZtLCS5REQBqBtZX1yezNk1Gq8h+iepc9FZQ3n++yV9F794vkQ=', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'connection': 'close', 'x-amz-request-id': '0942AFE1FECEC04D', 'date': 'Tue, 01 Aug 2017 22:58:35 GMT', 'content-type': 'application/xml'}, 'reason': 'Bad Request'}
DEBUG: S3Error: 400 (Bad Request)
DEBUG: HttpHeader: x-amz-region: us-east-1
DEBUG: HttpHeader: x-amz-id-2: 30gjCVw0WTi3ZAWRplPq9DuTyHZtLCS5REQBqBtZX1yezNk1Gq8h+iepc9FZQ3n++yV9F794vkQ=
DEBUG: HttpHeader: server: AmazonS3
DEBUG: HttpHeader: transfer-encoding: chunked
DEBUG: HttpHeader: connection: close
DEBUG: HttpHeader: x-amz-request-id: 0942AFE1FECEC04D
DEBUG: HttpHeader: date: Tue, 01 Aug 2017 22:58:35 GMT
DEBUG: HttpHeader: content-type: application/xml
DEBUG: object_get failed for './test.txt', deleting...
DEBUG: DeUnicodising u'./test.txt' using UTF-8
ERROR: S3 error: 400 (Bad Request)

dannyk81 · 2017-08-09T16:30:06Z

Ping :-)

Any ideas?

fviard · 2017-08-09T16:34:48Z

I have a little back log of issues to process but don't worry i will have a look. Le 9 août 2017 6:30 PM, "Danny Kulchinsky" <notifications@github.com> a écrit : Ping :-) Any ideas? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#905 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABAUpKidUo6DiRTJf4bJiqw8iIZwIutiks5sWd6PgaJpZM4OmBMx> .

dannyk81 · 2017-08-09T16:40:51Z

@fviard thank you!

fviard · 2017-08-12T00:04:49Z

Thank you for the detailed log.

I have an idea, is it possible for you to try a small change to see if it solves your issue?
1 line to add and 2 to slightly modify:

diff --git a/S3/ConnMan.py b/S3/ConnMan.py
index c1497bc..2a882df 100644
--- a/S3/ConnMan.py
+++ b/S3/ConnMan.py
@@ -209,8 +209,9 @@ class http_connection(object):
if ssl:
self.c = http_connection._https_connection(cfg.proxy_host, cfg.proxy_port)
debug(u'proxied HTTPSConnection(%s, %s)', cfg.proxy_host, cfg.proxy_port)
           self.c.set_tunnel(self.hostname, self.port)
           debug(u'tunnel to %s, %s', self.hostname, self.port)
           port = self.port and self.port or 443
           self.c.set_tunnel(self.hostname, port)
           debug(u'tunnel to %s, %s', self.hostname, port)
       else:
           self.c = httplib.HTTPConnection(cfg.proxy_host, cfg.proxy_port)
           debug(u'proxied HTTPConnection(%s, %s)', cfg.proxy_host, cfg.proxy_port)

I think that in the default case when no custom port set, then we create the tunnel with connect without specify the destination port. Thinking that the proxy will guess alone.
That may not be the case, and the proxy connecting to the port 80 instead of 443 (ssl) for example.

dannyk81 · 2017-08-12T01:30:55Z

Thanks for this!

Could you please post again the diff as above is not very clear.

dannyk81 · 2017-08-12T01:43:49Z

Okay, so I figured out what had to be changed (I think), this is the updated block:

            if ssl:
                self.c = http_connection._https_connection(cfg.proxy_host, cfg.proxy_port)
                debug(u'proxied HTTPSConnection(%s, %s)', cfg.proxy_host, cfg.proxy_port)
                port = self.port and self.port or 443
                self.c.set_tunnel(self.hostname, port)
                debug(u'tunnel to %s, %s', self.hostname, port)

And the good news! it works 👍 tested ls, put and get commands, all three work great!

dannyk81 · 2017-08-12T01:46:05Z

Here's my actual diff, just in case :)

diff --git a/S3/ConnMan.py b/S3/ConnMan.py
index c1497bc..2a882df 100644
--- a/S3/ConnMan.py
+++ b/S3/ConnMan.py
@@ -209,8 +209,9 @@ class http_connection(object):
             if ssl:
                 self.c = http_connection._https_connection(cfg.proxy_host, cfg.proxy_port)
                 debug(u'proxied HTTPSConnection(%s, %s)', cfg.proxy_host, cfg.proxy_port)
-                self.c.set_tunnel(self.hostname, self.port)
-                debug(u'tunnel to %s, %s', self.hostname, self.port)
+                port = self.port and self.port or 443
+                self.c.set_tunnel(self.hostname, port)
+                debug(u'tunnel to %s, %s', self.hostname, port)
             else:
                 self.c = httplib.HTTPConnection(cfg.proxy_host, cfg.proxy_port)
                 debug(u'proxied HTTPConnection(%s, %s)', cfg.proxy_host, cfg.proxy_port)

dannyk81 · 2017-08-15T02:29:01Z

Great! any plans to have a release anytime soon with this (and other) fixes?

We use Puppet + pip to deploy s3cmd, so would prefer an official release (instead of pulling from Git).

fviard · 2017-08-19T14:56:09Z

Yes, I hope to be able to do a new version in a few days.
A lot of issues have been found and fixed since the last release and the initial support of python 3.

dannyk81 · 2017-08-19T16:09:07Z

Great! looking forward to it :-)

…

On 19 August 2017 at 10:56, Florent Viard ***@***.***> wrote: Yes, I hope to be able to do a new version in a few days. A lot of issues have been found and fixed since the last release and the initial support of python 3. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#905 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AXJk7JePY4OZkQkI6WoG5e5Ng6WuzYt-ks5sZveLgaJpZM4OmBMx> .

dannyk81 · 2017-09-14T15:12:05Z

Any update about the next release 😊

fviard closed this as completed in 11d56a9 Aug 14, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

s3cmd v2.0.0 with Python 2.7.9 using SSL behind Squid Proxy, getting "ERROR: S3 error: 400 (Bad Request)" #905

s3cmd v2.0.0 with Python 2.7.9 using SSL behind Squid Proxy, getting "ERROR: S3 error: 400 (Bad Request)" #905

dannyk81 commented Jul 28, 2017 •

edited

Loading

dannyk81 commented Aug 1, 2017

dannyk81 commented Aug 1, 2017

dannyk81 commented Aug 9, 2017

fviard commented Aug 9, 2017 via email

dannyk81 commented Aug 9, 2017

fviard commented Aug 12, 2017

dannyk81 commented Aug 12, 2017

dannyk81 commented Aug 12, 2017 •

edited

Loading

dannyk81 commented Aug 12, 2017

dannyk81 commented Aug 15, 2017

fviard commented Aug 19, 2017

dannyk81 commented Aug 19, 2017 via email

dannyk81 commented Sep 14, 2017

s3cmd v2.0.0 with Python 2.7.9 using SSL behind Squid Proxy, getting "ERROR: S3 error: 400 (Bad Request)" #905

s3cmd v2.0.0 with Python 2.7.9 using SSL behind Squid Proxy, getting "ERROR: S3 error: 400 (Bad Request)" #905

Comments

dannyk81 commented Jul 28, 2017 • edited Loading

dannyk81 commented Aug 1, 2017

dannyk81 commented Aug 1, 2017

dannyk81 commented Aug 9, 2017

fviard commented Aug 9, 2017 via email

dannyk81 commented Aug 9, 2017

fviard commented Aug 12, 2017

dannyk81 commented Aug 12, 2017

dannyk81 commented Aug 12, 2017 • edited Loading

dannyk81 commented Aug 12, 2017

dannyk81 commented Aug 15, 2017

fviard commented Aug 19, 2017

dannyk81 commented Aug 19, 2017 via email

dannyk81 commented Sep 14, 2017

dannyk81 commented Jul 28, 2017 •

edited

Loading

dannyk81 commented Aug 12, 2017 •

edited

Loading