s3cmd sync fails with RemoteDisconnected #960

jimirocks · 2018-03-06T10:27:13Z

Looks like it's not able to fetch second part of listing the bucket content

DEBUG: Canonical Request:                                                                                                                                                                    
GET                                                                                                                                                                                          
/                                                                                                                                                                                            
marker=Foto%2F2001%2FGemini%2FSoftball%2FDownload%2FHow%20To%20Play%20Softball%285%29.htm&prefix=Foto%2F                                                                                     
host:pirkojm-backup.s3.amazonaws.com                                                                                                                                                         
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855                                                                                                        
x-amz-date:20180306T100045Z                                                                                                                                                                  
                                                                                                                                                                                             
host;x-amz-content-sha256;x-amz-date                                                                                                                                                         
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855                                                                                                                             
----------------------                                                                                                                                                                       
DEBUG: signature-v4 headers: {'x-amz-date': '20180306T100045Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=****/20180306/eu-west-1/s3/aws4_request,SignedHeaders=host;x-am
z-content-sha256;x-amz-date,Signature=****', 'x-amz-content-sha256': '****'}                                                                                                                                                                                      
DEBUG: Processing request, please wait...                                                                                                                                                    
DEBUG: get_hostname(pirkojm-backup): pirkojm-backup.s3.amazonaws.com                                                                                                                         
DEBUG: ConnMan.get(): re-using connection: https://pirkojm-backup.s3.amazonaws.com#1                                                                                                         
DEBUG: format_uri(): /?marker=Foto%2F2001%2FGemini%2FSoftball%2FDownload%2FHow%20To%20Play%20Softball%285%29.htm&prefix=Foto%2F                                                              
DEBUG: Sending request method_string='GET', uri='/?marker=Foto%2F2001%2FGemini%2FSoftball%2FDownload%2FHow%20To%20Play%20Softball%285%29.htm&prefix=Foto%2F', headers={'x-amz-date': '2018030
6T100045Z', 'Authorization': 'AWS4-HMAC-SHA256 Credential=****/20180306/eu-west-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=****', 'x-amz-content-sha256': '****'}, body=(0 bytes)                             
DEBUG: Response:                                                                                                                                                                             
{}

Here is the trace:

Problem: RemoteDisconnected: Remote end closed connection without response                                                                                                                   
S3cmd:   2.0.1                                                                                                                                                                               
python:   3.6.4 (default, Jan 25 2018, 15:20:29)                                                                                                                                             
[GCC 7.2.1 20180116]                                                                                                                                                                         
environment LANG=cs_CZ.UTF-8

Traceback (most recent call last):
  File "/usr/bin/s3cmd", line 3073, in <module>
    rc = main()
  File "/usr/bin/s3cmd", line 2989, in main
    rc = cmd_func(args)
  File "/usr/bin/s3cmd", line 1873, in cmd_sync
    return cmd_sync_local2remote(args)
  File "/usr/bin/s3cmd", line 1846, in cmd_sync_local2remote
    ret = _single_process(args[:-1])
  File "/usr/bin/s3cmd", line 1605, in _single_process
    return _child(destination_base, source_args)
  File "/usr/bin/s3cmd", line 1709, in _child
    remote_list, dst_exclude_list, remote_total_size = fetch_remote_list(destbase_with_source_list, recursive = True, require_attribs = True)
  File "/usr/lib/python3.6/site-packages/S3/FileLists.py", line 460, in fetch_remote_list
    objectlist, tmp_total_size = _get_filelist_remote(uri, recursive = True)
  File "/usr/lib/python3.6/site-packages/S3/FileLists.py", line 397, in _get_filelist_remote
    recursive = recursive, uri_params = uri_params)
  File "/usr/lib/python3.6/site-packages/S3/S3.py", line 308, in bucket_list
    for truncated, dirs, objects in self.bucket_list_streaming(bucket, prefix, recursive, uri_params, limit):
  File "/usr/lib/python3.6/site-packages/S3/S3.py", line 342, in bucket_list_streaming
    response = self.bucket_list_noparse(bucket, prefix, recursive, uri_params, max_keys)
  File "/usr/lib/python3.6/site-packages/S3/S3.py", line 373, in bucket_list_noparse
    response = self.send_request(request)
  File "/usr/lib/python3.6/site-packages/S3/S3.py", line 1256, in send_request
    http_response = conn.c.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/lib/python3.6/site-packages/S3/Custom_httplib3x.py", line 53, in httpresponse_patched_begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

The text was updated successfully, but these errors were encountered:

fviard · 2018-03-06T11:01:45Z

Thank you for the report, basically it is the same issue as:
#314
but a different exception with python3 and that skips retries.
I'm looking at how to fix the root issue

jimirocks · 2018-03-06T16:50:12Z

thanks for evaluation, in the meantime is there any workaround?

fviard · 2018-03-06T17:03:04Z

Using python2.
But i will try to push a workaround very fast. I'm looking into that.

Brullworfel · 2018-04-19T07:55:19Z

Hi! When it will be fixed?

fviard · 2018-06-03T23:44:58Z

Issue should now be fixed.

…ng with not crash As in python3 errno can be "None" with system raised exceptions, ensure that this case will not make s3cmd crash just for an error log output.

jrcarlsen · 2018-09-14T11:12:59Z

I am seeing a similar error:

DEBUG: get_hostname(backups): backups.s3.amazonaws.com
DEBUG: ConnMan.get(): re-using connection: https://backups.s3.amazonaws.com#81
DEBUG: format_uri(): /company/QE_Friday.7z
ERROR: Download of 'company/QE_Friday.7z' failed (Reason: Unknown OsError 0)
ERROR: Exiting now because of fatal error

Invoked as: /usr/local/bin/s3cmd sync --debug --delete-removed s3://backups/ /opt/fbrestore/mirrors/aws-mirror/
Problem: RemoteDisconnected: Remote end closed connection without response
S3cmd:   2.0.2
python:   3.6.5 (default, Apr  1 2018, 05:46:30)
[GCC 7.3.0]
environment LANG=C.UTF-8

Traceback (most recent call last):
  File "/usr/local/bin/s3cmd", line 3092, in <module>
    rc = main()
  File "/usr/local/bin/s3cmd", line 3001, in main
    rc = cmd_func(args)
  File "/usr/local/bin/s3cmd", line 1887, in cmd_sync
    return cmd_sync_remote2local(args)
  File "/usr/local/bin/s3cmd", line 1476, in cmd_sync_remote2local
    status, seq, size_transferred = _download(update_list, seq, remote_count + update_count, size_transferred, dir_cache)
  File "/usr/local/bin/s3cmd", line 1321, in _download
    response = s3.object_get(uri, dst_stream, dst_file, extra_label = seq_label)
  File "/usr/local/lib/python3.6/dist-packages/S3/S3.py", line 716, in object_get
    response = self.recv_file(request, stream, labels, start_position)
  File "/usr/local/lib/python3.6/dist-packages/S3/S3.py", line 1655, in recv_file
    http_response = conn.c.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/local/lib/python3.6/dist-packages/S3/Custom_httplib3x.py", line 53, in httpresponse_patched_begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

Let me know if I can provide more information that will help here.

jrcarlsen · 2018-09-14T11:24:25Z

Adding --skip-existing made the sync complete.

ajrnz · 2019-11-16T11:52:35Z

Same problem.

Adding --no-check-md5 worked for me.

Could it be that the server connection is timing out while the local server is building MD5 sums?

hopewise · 2020-09-16T05:54:03Z

I still have this issue, I tried the above suggestions, but did not work..

mattcasters · 2020-10-12T10:34:36Z

I still have this issue, I tried the above suggestions, but did not work..

Same here. Adding --no-check-md5 didn't solve the error. s3cmd version 2.0.2 on Ubuntu 10.04.

matepaiva · 2021-04-21T09:37:12Z

having the same problem here with get and with sync.

alitopaloglu · 2021-06-30T10:43:57Z

Still having the same issue, did not work with --no-check-md5 and --skip-existing params

INFO: Retrieving list of remote files for s3://blabla/c ...

Strange this is there is only 200 files under category named c but couldn't get the list of files from it.

pat-s · 2021-07-01T21:21:22Z

@alitopaloglu What worked for me was to use v2.1.0 as the release notes indicate a fix related to python3.8 which is most likely also being used on your side?

https://github.com/s3tools/s3cmd/releases

Maybe this could be highlighted somewhere or even the focal version could be bumped somehow.

fviard self-assigned this Mar 6, 2018

fviard added the to-be-fixed label Mar 6, 2018

fviard mentioned this issue Apr 30, 2018

Problem: <class 'TypeError: %d format: a number is required, not NoneType #967

Closed

fviard closed this as completed in b4f1424 Jun 3, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

s3cmd sync fails with RemoteDisconnected #960

s3cmd sync fails with RemoteDisconnected #960

jimirocks commented Mar 6, 2018

fviard commented Mar 6, 2018 •

edited

Loading

jimirocks commented Mar 6, 2018

fviard commented Mar 6, 2018

Brullworfel commented Apr 19, 2018

fviard commented Jun 3, 2018

jrcarlsen commented Sep 14, 2018

jrcarlsen commented Sep 14, 2018

ajrnz commented Nov 16, 2019

hopewise commented Sep 16, 2020

mattcasters commented Oct 12, 2020

matepaiva commented Apr 21, 2021

alitopaloglu commented Jun 30, 2021

pat-s commented Jul 1, 2021

s3cmd sync fails with RemoteDisconnected #960

s3cmd sync fails with RemoteDisconnected #960

Comments

jimirocks commented Mar 6, 2018

fviard commented Mar 6, 2018 • edited Loading

jimirocks commented Mar 6, 2018

fviard commented Mar 6, 2018

Brullworfel commented Apr 19, 2018

fviard commented Jun 3, 2018

jrcarlsen commented Sep 14, 2018

jrcarlsen commented Sep 14, 2018

ajrnz commented Nov 16, 2019

hopewise commented Sep 16, 2020

mattcasters commented Oct 12, 2020

matepaiva commented Apr 21, 2021

alitopaloglu commented Jun 30, 2021

pat-s commented Jul 1, 2021

fviard commented Mar 6, 2018 •

edited

Loading