Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

s3 incorrectly handles whitespace in file name #706

Closed
fedorl opened this issue Mar 19, 2014 · 8 comments
Closed

s3 incorrectly handles whitespace in file name #706

fedorl opened this issue Mar 19, 2014 · 8 comments
Labels
bug This issue is a bug. s3sync

Comments

@fedorl
Copy link

fedorl commented Mar 19, 2014

Upload file with white space in key, e.g. "test space.txt"

sync parent directory s3->s3, will result in NoSuchKey error
broken both in 1.3.1 and development

1.3.1:
test space.txt A client error (NoSuchKey) occurred when calling the CopyObject operation

development:
test+space.txt A client error (NoSuchKey) occurred when calling the CopyObject operation
@fedorl fedorl changed the title s3 does incorrectly handles whitespace in file name s3 incorrectly handles whitespace in file name Mar 19, 2014
@danielgtaylor
Copy link
Contributor

I can confirm this issue. Steps to repro:

$ mkdir tmp
$ echo "test" >"tmp/test space.txt"
$ aws s3 sync tmp s3://your-bucket
upload: tmp/test space.txt to s3://your-bucket/test space.txt
$ mkdir tmp2
$ aws --debug s3 sync s3://your-bucket tmp2
2014-03-19 13:35:28,439 - botocore.service - DEBUG - Creating service object for: s3
2014-03-19 13:35:28,440 - botocore.hooks - DEBUG - Event service-data-loaded.s3: calling handler <function signature_overrides at 0x10afea398>
2014-03-19 13:35:28,440 - botocore.hooks - DEBUG - Event service-created: calling handler <function register_retries_for_service at 0x10afea1b8>
2014-03-19 13:35:28,440 - botocore.handlers - DEBUG - Registering retry handlers for service: Service(s3)
2014-03-19 13:35:28,440 - awscli.customizations.s3.executor - DEBUG - Submitting task: <awscli.customizations.s3.tasks.BasicTask object at 0x10b2dff90>
2014-03-19 13:35:28,440 - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 11)
2014-03-19 13:35:28,440 - awscli.customizations.s3.executor - DEBUG - Worker thread invoking task: <awscli.customizations.s3.tasks.BasicTask object at 0x10b2dff90>
2014-03-19 13:35:28,441 - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 11)
2014-03-19 13:35:28,441 - botocore.operation - DEBUG - Operation:GetObject called with kwargs: {'bucket': u'your-bucket', 'key': u'test+space.txt'}
2014-03-19 13:35:28,441 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2014-03-19 13:35:28,441 - botocore.operation - DEBUG - Creating parameter objects for: Operation:GetObject
2014-03-19 13:35:28,441 - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 11)
2014-03-19 13:35:28,444 - botocore.endpoint - DEBUG - Making request for Operation:GetObject (verify_ssl=True) with params: {'headers': {}, 'uri_params': {u'Bucket': u'your-bucket', u'Key': u'test+space.txt'}, 'payload': <botocore.payload.XMLPayload object at 0x10b40d610>}
2014-03-19 13:35:28,442 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2014-03-19 13:35:28,445 - botocore.endpoint - DEBUG - Building URI for rest endpoint.
2014-03-19 13:35:28,444 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2014-03-19 13:35:28,445 - botocore.endpoint - DEBUG - Templated URI path: /{Bucket}/{Key}
2014-03-19 13:35:28,445 - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 11)
2014-03-19 13:35:28,445 - botocore.endpoint - DEBUG - Templated URI query_params: versionId={VersionId}&response-content-type={ResponseContentType}&response-content-language={ResponseContentLanguage}&response-expires={ResponseExpires}&response-cache-control={ResponseCacheControl}&response-content-disposition={ResponseContentDisposition}&response-content-encoding={ResponseContentEncoding}
2014-03-19 13:35:28,445 - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 11)
2014-03-19 13:35:28,446 - botocore.endpoint - DEBUG - Rendered path: /your-bucket/test%2Bspace.txt
2014-03-19 13:35:28,446 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2014-03-19 13:35:28,446 - botocore.endpoint - DEBUG - Rendered query_params: 
2014-03-19 13:35:28,446 - botocore.hooks - DEBUG - Event before-auth.s3: calling handler <function fix_s3_host at 0x10afea0c8>
2014-03-19 13:35:28,447 - botocore.handlers - DEBUG - Checking for DNS compatible bucket for: https://s3.amazonaws.com/your-bucket/test%2Bspace.txt
2014-03-19 13:35:28,447 - botocore.handlers - DEBUG - URI updated to: https://your-bucket.s3.amazonaws.com/test%2Bspace.txt
2014-03-19 13:35:28,447 - botocore.auth - DEBUG - Calculating signature using hmacv1 auth.
2014-03-19 13:35:28,446 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2014-03-19 13:35:28,446 - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 11)
2014-03-19 13:35:28,447 - botocore.auth - DEBUG - HTTP request method: GET
2014-03-19 13:35:28,448 - botocore.auth - DEBUG - StringToSign:
GET


Wed, 19 Mar 2014 20:35:28 GMT
/your-bucket/test%2Bspace.txt
2014-03-19 13:35:28,448 - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 11)
2014-03-19 13:35:28,449 - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [GET]>
2014-03-19 13:35:28,449 - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 11)
2014-03-19 13:35:28,449 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2014-03-19 13:35:28,449 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2014-03-19 13:35:28,449 - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 11)
2014-03-19 13:35:28,450 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2014-03-19 13:35:28,450 - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 11)
2014-03-19 13:35:28,450 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2014-03-19 13:35:28,451 - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-3, started daemon 4506808320)>
2014-03-19 13:35:28,564 - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "GET /test%2Bspace.txt HTTP/1.1" 404 None
2014-03-19 13:35:28,565 - botocore.response - DEBUG - Response Headers:
x-amz-id-2: wDgA4efV1JEpXUe0fqJNHUerRxiVFbCh06xc8Zcq6TQsXUxqm4c/9tJAslOjM0gv
server: AmazonS3
transfer-encoding: chunked
x-amz-request-id: 48A4BE87AF96F3E0
date: Wed, 19 Mar 2014 20:35:25 GMT
content-type: application/xml
2014-03-19 13:35:28,565 - botocore.response - DEBUG - Response Body:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>test+space.txt</Key><RequestId>48A4BE87AF96F3E0</RequestId><HostId>wDgA4efV1JEpXUe0fqJNHUerRxiVFbCh06xc8Zcq6TQsXUxqm4c/9tJAslOjM0gv</HostId></Error>
2014-03-19 13:35:28,567 - botocore.hooks - DEBUG - Event needs-retry.s3.GetObject: calling handler <botocore.retryhandler.RetryHandler object at 0x10b2df490>
2014-03-19 13:35:28,568 - botocore.retryhandler - DEBUG - No retry needed.
2014-03-19 13:35:28,568 - botocore.hooks - DEBUG - Event after-call.s3.GetObject: calling handler <awscli.errorhandler.ErrorHandler object at 0x10b2c5ed0>
2014-03-19 13:35:28,568 - awscli.errorhandler - DEBUG - HTTP Response Code: 404
2014-03-19 13:35:28,568 - awscli.customizations.s3.tasks - DEBUG - A client error (NoSuchKey) occurred when calling the GetObject operation: The specified key does not exist.
Traceback (most recent call last):
  File "/Users/taylordt/Projects/Amazon/aws-cli/awscli/customizations/s3/tasks.py", line 88, in _execute_task
    getattr(filename, filename.operation_name)()
  File "/Users/taylordt/Projects/Amazon/aws-cli/awscli/customizations/s3/fileinfo.py", line 231, in download
    response_data, http = operate(self.service, 'GetObject', params)
  File "/Users/taylordt/Projects/Amazon/aws-cli/awscli/customizations/s3/utils.py", line 153, in operate
    http_response, response_data = operation.call(**kwargs)
  File "/Users/taylordt/Projects/Amazon/aws-cli/venv/lib/python2.7/site-packages/botocore/operation.py", line 74, in call
    parsed=response[1])
  File "/Users/taylordt/Projects/Amazon/aws-cli/venv/lib/python2.7/site-packages/botocore/session.py", line 617, in emit
    return self._events.emit(event_name, **kwargs)
  File "/Users/taylordt/Projects/Amazon/aws-cli/venv/lib/python2.7/site-packages/botocore/hooks.py", line 150, in emit
    response = handler(**kwargs)
  File "/Users/taylordt/Projects/Amazon/aws-cli/awscli/errorhandler.py", line 75, in __call__
    http_status_code=http_response.status_code)
ClientError: A client error (NoSuchKey) occurred when calling the GetObject operation: The specified key does not exist.
2014-03-19 13:35:28,646 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in worker thread, shutting down worker thread.
2014-03-19 13:35:28,646 - awscli.customizations.s3.executor - DEBUG - Received print task: {'message': u'download failed: s3://your-bucket/test+space.txt to tmp2/test+space.txt A client error (NoSuchKey) occurred when calling the GetObject operation: The specified key does not exist.', 'error': True}
download failed: s3://your-bucket/test+space.txt to tmp2/test+space.txt A client error (NoSuchKey) occurred when calling the GetObject operation: The specified key does not exist.
2014-03-19 13:35:28,669 - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-3, stopped daemon 4506808320)>
2014-03-19 13:35:28,670 - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-4, stopped daemon 4511014912)>
2014-03-19 13:35:28,670 - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-4, stopped daemon 4511014912)>
2014-03-19 13:35:28,670 - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-5, stopped daemon 4515221504)>
2014-03-19 13:35:28,670 - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-5, stopped daemon 4515221504)>
2014-03-19 13:35:28,670 - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-6, stopped daemon 4519428096)>
2014-03-19 13:35:28,670 - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-6, stopped daemon 4519428096)>
2014-03-19 13:35:28,671 - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-7, stopped daemon 4523634688)>
2014-03-19 13:35:28,671 - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-7, stopped daemon 4523634688)>
2014-03-19 13:35:28,671 - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-8, stopped daemon 4527841280)>
2014-03-19 13:35:28,671 - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-8, stopped daemon 4527841280)>
2014-03-19 13:35:28,671 - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-9, stopped daemon 4532047872)>
2014-03-19 13:35:28,671 - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-9, stopped daemon 4532047872)>
2014-03-19 13:35:28,671 - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-10, stopped daemon 4536254464)>
2014-03-19 13:35:28,672 - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-10, stopped daemon 4536254464)>
2014-03-19 13:35:28,672 - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-11, stopped daemon 4540461056)>
2014-03-19 13:35:28,672 - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-11, stopped daemon 4540461056)>
2014-03-19 13:35:28,672 - awscli.customizations.s3.executor - DEBUG - Waiting for thread to shutdown: <Worker(Thread-12, stopped daemon 4544667648)>
2014-03-19 13:35:28,672 - awscli.customizations.s3.executor - DEBUG - Thread has been shutdown: <Worker(Thread-12, stopped daemon 4544667648)>
2014-03-19 13:35:28,672 - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for result thread.
2014-03-19 13:35:28,672 - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for IO thread.
2014-03-19 13:35:28,673 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in IO thread, shutting down.
2014-03-19 13:35:28,673 - awscli.customizations.s3.executor - DEBUG - Waiting for result thread to shutdown.
2014-03-19 13:35:28,673 - awscli.customizations.s3.executor - DEBUG - Shutdown request received in print thread, shutting down print thread.
2014-03-19 13:35:28,674 - awscli.customizations.s3.executor - DEBUG - Waiting for IO thread to shutdown.
2014-03-19 13:35:28,674 - awscli.customizations.s3.executor - DEBUG - All threads have been shutdown.

@JRyanJones
Copy link

I can confirm this is still an issue in the 1.3.2 release. Sync uploads files correctly but tries to download with spaces converted to '+', yielding non-existent key errors.

@anazar
Copy link

anazar commented Mar 27, 2014

+1 also having this issue.

@jamesls
Copy link
Member

jamesls commented Mar 27, 2014

This should be fixed here (boto/botocore#264). I just need a code review from @danielgtaylor before merging in the fix. Also added integ tests for this here to ensure we don't regress on this.

@jamesls
Copy link
Member

jamesls commented Mar 27, 2014

Fix has been merged, this will go out in the next CLI release. Thanks everyone.

@jamesls jamesls closed this as completed Mar 27, 2014
@jamesls jamesls mentioned this issue Apr 1, 2014
Closed
@johnnylyne
Copy link

I am still seeing this issue using Node AWS SDK to process a file upload to an S3 bucket within an AWS Lambda. The lambda is triggered by a file upload to an S3 bucket. When the file uploaded contains whitespace the event.records[].S3.Object.key contains + instead of space.

thoward-godaddy pushed a commit to thoward-godaddy/aws-cli that referenced this issue Feb 12, 2022
@sliedig @jfuss
docs: Updated virtualenv alias name for 3.7 in guide. (aws#706)
ca4e35c
@algermakiputin
Copy link

As of today still seeing this issue when triggering s3 upload file to AWS lambda, for now will replace the plus sign to white space to solve this issue

@AbelGuitian
Copy link

As of today still seeing this issue when triggering s3 upload file to AWS lambda, for now will replace the plus sign to white space to solve this issue

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug This issue is a bug. s3sync
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@danielgtaylor @anazar @jamesls @JRyanJones @fedorl @johnnylyne @algermakiputin @AbelGuitian