Regression: Endpoint URL not honored in s3 commands

[kwinters]

In #549 e27bbc6 the fix_s3_host handler is unregistered if --endpoint-url is provided.

But with the latest version this behavior is happening again, resulting in credential failures (the provided access key does not exist at s3.amazonaws.com).

Workaround: Use a bucket name with a period in it, which also skips the fix_s3_host behavior.

$ aws --debug --profile=twosite s3api --region us-west-1 --endpoint-url https://10.224.3.179:8082 --no-verify-ssl create-bucket --bucket "zomg"
2015-02-11 16:03:56,852 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.7.6 Python/2.7.6 Linux/3.13.0-35-generic, botocore version: 0.87.0
2015-02-11 16:03:56,852 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x7f0e167f3b90>
2015-02-11 16:03:56,852 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider at 0x7f0e16aadc08>
2015-02-11 16:03:56,853 - MainThread - botocore.service - DEBUG - Creating service object for: s3
2015-02-11 16:03:56,886 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.s3: calling handler <function register_retries_for_service at 0x7f0e16f8d2a8>
2015-02-11 16:03:56,889 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: s3
2015-02-11 16:03:56,890 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.s3: calling handler <function register_retries_for_service at 0x7f0e16f8d2a8>
2015-02-11 16:03:56,890 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: s3
2015-02-11 16:03:56,890 - MainThread - botocore.service - DEBUG - Creating operation objects for: Service(s3)
2015-02-11 16:03:56,895 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3api: calling handler <function add_waiters at 0x7f0e16a3f9b0>
2015-02-11 16:03:56,898 - MainThread - awscli.clidriver - DEBUG - OrderedDict([(u'acl', <awscli.arguments.CLIArgument object at 0x7f0e165942d0>), (u'bucket', <awscli.arguments.CLIArgument object at 0x7f0e165945d0>), (u'create-bucket-configuration', <awscli.arguments.CLIArgument object at 0x7f0e16594610>), (u'grant-full-control', <awscli.arguments.CLIArgument object at 0x7f0e16594650>), (u'grant-read', <awscli.arguments.CLIArgument object at 0x7f0e16594690>), (u'grant-read-acp', <awscli.arguments.CLIArgument object at 0x7f0e165946d0>), (u'grant-write', <awscli.arguments.CLIArgument object at 0x7f0e16594750>), (u'grant-write-acp', <awscli.arguments.CLIArgument object at 0x7f0e16594310>)])
2015-02-11 16:03:56,898 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.create-bucket: calling handler <function add_streaming_output_arg at 0x7f0e16b8e140>
2015-02-11 16:03:56,898 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.create-bucket: calling handler <function add_cli_input_json at 0x7f0e16aa8e60>
2015-02-11 16:03:56,899 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.create-bucket: calling handler <function unify_paging_params at 0x7f0e16b8ef50>
2015-02-11 16:03:56,899 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.create-bucket: calling handler <function add_generate_skeleton at 0x7f0e16aad410>
2015-02-11 16:03:56,899 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.s3api.create-bucket: calling handler <bound method CliInputJSONArgument.override_required_args of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x7f0e16594790>>
2015-02-11 16:03:56,900 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.s3api.create-bucket: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7f0e167bbf90>>
2015-02-11 16:03:56,901 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.create-bucket.acl: calling handler <function uri_param at 0x7f0e16bcc9b0>
2015-02-11 16:03:56,902 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.create-bucket.bucket: calling handler <function uri_param at 0x7f0e16bcc9b0>
2015-02-11 16:03:56,902 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.s3.create-bucket: calling handler <awscli.argprocess.ParamShorthand object at 0x7f0e167fd590>
2015-02-11 16:03:56,902 - MainThread - awscli.argprocess - DEBUG - Detected structure: scalar
2015-02-11 16:03:56,902 - MainThread - awscli.arguments - DEBUG - Unpacked value of u'zomg' for parameter "bucket": u'zomg'
2015-02-11 16:03:56,902 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.create-bucket.create-bucket-configuration: calling handler <function uri_param at 0x7f0e16bcc9b0>
2015-02-11 16:03:56,902 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.create-bucket.grant-full-control: calling handler <function uri_param at 0x7f0e16bcc9b0>
2015-02-11 16:03:56,903 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.create-bucket.grant-read: calling handler <function uri_param at 0x7f0e16bcc9b0>
2015-02-11 16:03:56,903 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.create-bucket.grant-read-acp: calling handler <function uri_param at 0x7f0e16bcc9b0>
2015-02-11 16:03:56,903 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.create-bucket.grant-write: calling handler <function uri_param at 0x7f0e16bcc9b0>
2015-02-11 16:03:56,903 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.create-bucket.grant-write-acp: calling handler <function uri_param at 0x7f0e16bcc9b0>
2015-02-11 16:03:56,903 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.create-bucket.cli-input-json: calling handler <function uri_param at 0x7f0e16bcc9b0>
2015-02-11 16:03:56,904 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.create-bucket.generate-cli-skeleton: calling handler <function uri_param at 0x7f0e16bcc9b0>
2015-02-11 16:03:56,904 - MainThread - botocore.hooks - DEBUG - Event calling-command.s3api.create-bucket: calling handler <bound method GenerateCliSkeletonArgument.generate_json_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7f0e167bbf90>>
2015-02-11 16:03:56,904 - MainThread - botocore.hooks - DEBUG - Event calling-command.s3api.create-bucket: calling handler <bound method CliInputJSONArgument.add_to_call_parameters of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x7f0e16594790>>
2015-02-11 16:03:56,904 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2015-02-11 16:03:56,904 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2015-02-11 16:03:56,904 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2015-02-11 16:03:56,905 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2015-02-11 16:03:56,907 - MainThread - botocore.operation - DEBUG - Operation:CreateBucket called with kwargs: {u'Bucket': u'zomg'}
2015-02-11 16:03:56,908 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.s3: calling handler <function register_retries_for_service at 0x7f0e16f8d2a8>
2015-02-11 16:03:56,909 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: s3
2015-02-11 16:03:56,909 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.CreateBucket: calling handler <function add_expect_header at 0x7f0e16f8d488>
2015-02-11 16:03:56,909 - MainThread - botocore.endpoint - DEBUG - Making request for <botocore.model.OperationModel object at 0x7f0e16604a10> (verify_ssl=False) with params: {'query_string': {}, 'headers': {}, 'url_path': u'/zomg', 'body': '', 'method': u'PUT'}
2015-02-11 16:03:56,909 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.CreateBucket: calling handler <function request_created at 0x7f0e16627410>
2015-02-11 16:03:56,910 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.CreateBucket: calling handler <function fix_s3_host at 0x7f0e16f8d140>
2015-02-11 16:03:56,910 - MainThread - botocore.handlers - DEBUG - Checking for DNS compatible bucket for: https://10.224.3.179:8082/zomg
2015-02-11 16:03:56,910 - MainThread - botocore.handlers - DEBUG - URI updated to: https://zomg.s3.amazonaws.com
2015-02-11 16:03:56,910 - MainThread - botocore.auth - DEBUG - Calculating signature using hmacv1 auth.
2015-02-11 16:03:56,910 - MainThread - botocore.auth - DEBUG - HTTP request method: PUT
2015-02-11 16:03:56,910 - MainThread - botocore.auth - DEBUG - StringToSign:
PUT

Wed, 11 Feb 2015 21:03:56 GMT
/zomg/
2015-02-11 16:03:56,917 - MainThread - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [PUT]>
2015-02-11 16:03:56,918 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): zomg.s3.amazonaws.com
/usr/local/lib/python2.7/dist-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:734: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
2015-02-11 16:03:57,270 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "PUT / HTTP/1.1" 403 None
2015-02-11 16:03:57,273 - MainThread - botocore.parsers - DEBUG - Response headers:
{'content-type': 'application/xml',
'date': 'Wed, 11 Feb 2015 21:03:58 GMT',
'server': 'AmazonS3',
'transfer-encoding': 'chunked',
'x-amz-id-2': 'yLdQVz8LACrfbBxuM3Pp7nnBQ4dTCEhSmlhxa15wICdWZxgW9rKqBTcyt6Qxnv0g',
'x-amz-request-id': '5E38794F0631EB1C'}
2015-02-11 16:03:57,273 - MainThread - botocore.parsers - DEBUG - Response body:

InvalidAccessKeyIdThe AWS Access Key Id you provided does not exist in our records.PPY2K5EX5O7AI5E8DQGA5E38794F0631EB1CyLdQVz8LACrfbBxuM3Pp7nnBQ4dTCEhSmlhxa15wICdWZxgW9rKqBTcyt6Qxnv0g
2015-02-11 16:03:57,274 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.CreateBucket: calling handler <botocore.retryhandler.RetryHandler object at 0x7f0e16604890>
2015-02-11 16:03:57,274 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2015-02-11 16:03:57,275 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.CreateBucket: calling handler <function enhance_error_msg at 0x7f0e16aa86e0>
2015-02-11 16:03:57,275 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.CreateBucket: calling handler <awscli.errorhandler.ErrorHandler object at 0x7f0e167fd5d0>
2015-02-11 16:03:57,275 - MainThread - awscli.errorhandler - DEBUG - HTTP Response Code: 403
2015-02-11 16:03:57,275 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 197, in main
return command_table[parsed_args.command](remaining, parsed_args)
File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 373, in call
return command_table[parsed_args.operation](remaining, parsed_globals)
File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 531, in call
self._operation_object, call_parameters, parsed_globals)
File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 636, in invoke
**parameters)
File "/usr/local/lib/python2.7/dist-packages/botocore/operation.py", line 176, in call
parsed=response[1])
File "/usr/local/lib/python2.7/dist-packages/botocore/session.py", line 735, in emit
return self._events.emit(event_name, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 226, in emit
return self._emit(event_name, kwargs)
File "/usr/local/lib/python2.7/dist-packages/botocore/hooks.py", line 209, in _emit
response = handler(**kwargs)
File "/usr/local/lib/python2.7/dist-packages/awscli/errorhandler.py", line 70, in call
http_status_code=http_response.status_code)
ClientError: A client error (InvalidAccessKeyId) occurred when calling the CreateBucket operation: The AWS Access Key Id you provided does not exist in our records.
2015-02-11 16:03:57,277 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

A client error (InvalidAccessKeyId) occurred when calling the CreateBucket operation: The AWS Access Key Id you provided does not exist in our records.

[jamesls]

Confirmed, I ran into this as well. More debug info:

This happened somewhere inbetween the 1.7.4 to 1.7.5 range which was when we made a lot of internal auth changes. Likely somewhere in botocore: boto/botocore@0.85.0...0.86.0

1.7.4

Notice we're connecting to localhost, which is the expected behavior.

~ $ pip install --force awscli==1.7.4
~ $ aws s3 ls s3://foo/ --endpoint-url http://localhost/ --debug 2>&1 | grep "Starting new HTTP"
2015-02-21 10:49:49,933 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTP connection (1): localhost

1.7.5

Notice we're connecting to foo.s3.amazonaws.com, which is not the expected behavior.

~ $ pip install --force awscli==1.7.5
~ $ aws s3 ls s3://foo/ --endpoint-url http://localhost/ --debug 2>&1 | grep "Starting new HTTP"
2015-02-21 10:50:31,239 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTP connection (1): foo.s3.amazonaws.com

I've also confirmed the incorrect behavior still exists in the latest 1.7.11 version.

cc @kyleknap @danielgtaylor