Creating of SQS queues with latest botorecore (1.29.127) is broken

[PierreKiwi]

Hello moto team,

Looks like moto is broken with the latest botocore (1.29.127) as AWS has changed the format for interacting with some services (includig SQS). When creating a SQS queue, it returns {} rather than a Queue object (I use a SQS resource).

I have opened an issue in their tracker (see boto/botocore#2930) but it looks like moto might need to make some changes too.

You can reproduce the issue with:

import moto
import boto3
from uuid import uuid4

@moto.mock_sqs
def test_create_queue():
    sqs = boto3.resource("sqs", region_name="us-east-1")

    q_name = str(uuid4())[0:6]
    new_queue = sqs.create_queue(QueueName=q_name)

    print(new_queue)

test_create_queue()

With pip install boto3==1.26.126 botocore==1.29.126 moto, the output is sqs.Queue(url='https://sqs.us-east-1.amazonaws.com/123456789012/a94c6c').

With pip install boto3==1.26.127 botocore==1.29.127 moto, the output is {}.

[kellertk]

AWS has changed the format for interacting with some services

Just wanted to clarify on this point. AWS has only changed the wire format for SQS. Changing the wire protocol for a service is a somewhat large undertaking and we don't do it often. The SQS service itself supports both the old and new API formats so that old SDK clients will continue to work, but new versions of boto* will only ever talk to SQS in the new format.

[JayThomason]

We were just bit by this as well and I used an almost identical test script haha.

My testing seemed to to indicate that boto3 >= 1.26.0 does not work with any versions of moto, while boto3 1.25.X did work, although maybe the issue was in botocore -- I didn't check the version of that.

This is pretty easy to replicate by spinning up a python:3.11 docker container, installing moto and boto3, running the test script, and then switching between different versions of boto3.

[SamStephens]

@JayThomason it's botocore - what you're seeing is that boto3 1.26.x will use any available version of botocore that is > 1.29.x and < 1.30.0, whereas boto3 1.15.x will use any available version of botocore that is > 1.28.x and < 1.29.0.

So any install of boto3 1.26.x will pick up botocore==1.29.127 (or later) which has the breaking change, whereas any install of boto3 1.25.x will pick up botocore==1.18.5 and work.

This was gnarly for me, because all of my requirements.txt specification are on fixed versions, so I was surprised to see builds suddenly failing with no changes made.

[ppena-LiveData]

It looks like botocore made a major change to botocore/data/sqs/2012-11-05/service-2.json earlier today. The responses from the SQS service are no longer XML and are now JSON. Since moto/sqs/responses.py has the responses as XML, that (and maybe other moto files) will have to change.

[JayThomason]

@SamStephens Yes, it turns out that the issue with my project in particular is that it was pinning the version of boto3 but not the version of botocore.

[mikegrima]

Which versions supports JSON responses? If we were to swap out all the XML for JSON in moto, what impact will this have on older versions of botocore? We really don't want to push a dependency management mess on everyone.

More of a question for @kellertk

[bblommers]

@mikegrima The other SDK's presumably still only support XML, so I feel like we should be supporting both formats.

The question in my mind is more - how do we choose which format to return? Can we simply use the ContentType header?

I should be able to look into this next week.

[ppena-LiveData]

I ran an experiment, and I see that moto.core.botocore_stubber.BotocoreStubber.__call__() is now given a request with a Content-Type header of application/x-amz-json-1.0, whereas the previous botocore was sending a request with a Content-Type header of application/x-www-form-urlencoded; charset=utf-8.

It probably doesn't matter, but there's also now an added X-Amz-Target header, which in my case had AmazonSQS.GetQueueUrl (since I was testing get_queue_by_name(), not create_queue()).

[kellertk]

@mikegrima We'v reverted changes to SQS in botocore 1.29.129.

I think @nateprewitt would have more recommendations on how to manage the different version problem.

[bblommers]

The PR that contains the revert has more information: boto/botocore#2931

Seeing that this format-change in SQS will be re-introduced in the feature, Moto will still need to support it. Marking it as an (lower-priority) enhancement to do so.

[nateprewitt]

@mikegrima

The question in my mind is more - how do we choose which format to return? Can we simply use the ContentType header?

Yes, the Content-Type header would be a reasonable way to determine which response format is valid. Older SDKs will continue to send application/x-www-form-urlencoded which is what receives the current XML responses. Newer SDKs will start sending application/x-amz-json-1.0 once this change is rolled out again. This expects a JSON payload in the response.

Edit: botocore==1.29.127 will continue to operate against production SQS in the new format if you need a reference client. Requests should be "round-trippable" at this time.

[mikegrima]

I wonder if what we could do in the very short term is just override the header value for all SQS requests (moto side)? Not ideal, but it should buy us some time.

[SamStephens]

As part of addressing this, would it be worth adding an general assertion/set of assertions as to the response format returned by botocode?

If there had been an assertion that the Content-Type header was application/x-www-form-urlencoded in place, when botocore 1.29.127 introduced the breaking change, we would have seen our tests fail because the assertion was violated, rather than the obscure empty response we saw.

[ppena-LiveData]

I wonder if what we could do in the very short term is just override the header value for all SQS requests (moto side)? Not ideal, but it should buy us some time.

@mikegrima, the problem is that _do_get_response() in botocore/endpoint.py looks at operation_model to know whether it's expecting JSON or XML, so changing the header of the request/response wouldn't help.

[bblommers]

Moto >= 4.1.11.dev7 now also supports both JSON and XML responses. It will return the appropriate format based on the ContentType-header.

So when AWS re-enables this feature, we should be ready.

[nateprewitt]

Thank you for the quick turnaround, @bblommers! Really appreciate it.