Error when creating SQS queues with moto

[PierreKiwi]

Describe the bug

With latest version of botocore (1.29.127), my unit tests started to fail when creating a fake SQS queue with moto.

It works fine with 1.29.126 though so I suspect 3e1f77d could be the cause of it.

Expected Behavior

The fake queue should be created as expected.

Current Behavior

aws_resource_sqs.create_queue(QueueName="OurFakeQueue") returns an empty dictionary rather than a Queue object.

Reproduction Steps

import moto
import boto3
from uuid import uuid4

@moto.mock_sqs
def test_create_queue():
    sqs = boto3.resource("sqs", region_name="us-east-1")

    q_name = str(uuid4())[0:6]
    new_queue = sqs.create_queue(QueueName=q_name)

    print(new_queue)

test_create_queue()

With pip install boto3==1.26.126 botocore==1.29.126 moto, the output is sqs.Queue(url='https://sqs.us-east-1.amazonaws.com/123456789012/a94c6c')

With pip install boto3==1.26.127 botocore==1.29.127 moto, the output is {}

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.29.127

Environment details (OS name and version, etc.)

Darwin lmi-macbookpro183.local 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar 6 20:59:28 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000 arm64

[nateprewitt]

Hi @PierreKiwi, thanks for reaching out. SQS did in fact change the wire protocol it uses in today's release. The service itself still supports the old and new format for clients, but going forward new releases of Boto3 will only speak to it over a JSON RPC format.

My hunch is that moto has hardcoded their SQS mocking to use the old wire format. That should still work with older copies of Boto3, but will need to be updated going forward. For next steps, this should be raised with moto library so it can be addressed for future releases.

[PierreKiwi]

That makes sense. Wanted to report the problem here first as I was not sure how to interpret the format changes. Will open an issue in their tracker (I expect a lot of unit tests to start failing as moto automatically uses the latest version of boto3/botocore).

Thanks a lot for your swift response.

[ArzelaAscoIi]

We are getting the same errors for 127, but 1.29.126 works. We are using localstack and are getting this error:

An error occurred (500) when calling the GetQueueUrl operation (reached max retries: 0): <?xml version='1.0' encoding='utf-8'?>
haystack-hub-api-file-ingestion-consumer-1  | <ErrorResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><Error><Code>InternalError</Code><Message>exception while calling sqs with unknown operation: Operation detection failed. Missing Action in request for query-protocol service ServiceModel(sqs).</Message></Error><RequestId>3SZ5YGRU1KOGU5HXLXJYR2EIRIS1ZFPIT9H3Q2JME3NOCJVKMI0Q</RequestId></ErrorResponse>

[dikiigr]

We getting the same error, but we use client. This error can be reproduced with this code:

import boto3
import moto

with moto.mock_sqs():
    sqs = boto3.client("sqs", region_name="eu-west-2")
    create_resp = sqs.create_queue(QueueName="test")
    print("QueueUrl" in create_resp)

We expect QueueUrl property to be in response as in documentation (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs/client/create_queue.html#SQS.Client.create_queue)

Versions:

Python 3.10.10
botocore 1.29.127
moto 4.1.6
boto3 1.26.126

[Clokkehl]

We're getting the same error when using client, reproduced:

...
with moto.mock_sqs():
    sqs_client = mock_config["sqs_client"]
    response = sqs_client.create_queue(QueueName=mock_config["queue_name"])
    queue_url = response["QueueUrl"]

As people have pointed out, expected QueueUrl in response. HTTP headers point to a 200 so the response isn't failing.

Versions:
Python 3.9.16
Botocore 1.29.127
moto 4.1.8
boto3 1.26.127

[Ealameda31]

I can call the queue fine with AWS using the request but moto can't find it in its session the test_queue is created but the url no longer points to the path of the queue.

[joaomaranhao]

We getting the same error, but we use client. This error can be reproduced with this code:

import boto3
import moto

with moto.mock_sqs():
    sqs = boto3.client("sqs", region_name="eu-west-2")
    create_resp = sqs.create_queue(QueueName="test")
    print("QueueUrl" in create_resp)

We expect QueueUrl property to be in response as in documentation (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs/client/create_queue.html#SQS.Client.create_queue)

Versions:

Python 3.10.10 botocore 1.29.127 moto 4.1.6 boto3 1.26.126

Yes, this issue doesn't even have to do with moto
After the update of botocore to 1.29.127, create_queue is not returning "QueueUrl" as it should
If you downgrade botocore to 1.29.115, the code will work again

[nateprewitt]

Yes, this issue doesn't even have to do with moto
After the update of botocore to 1.29.127, create_queue is not returning "QueueUrl" as it should

Could you clarify this a bit more @joaomaranhao? Are you saying live calls to SQS (not using moto) are no longer returning QueueUrl?

[Ealameda31]

@nateprewitt internally AWS url in queues are now request-based and not queue-based.

So instead of sqs.Queue(url="https://sqs.us-west-2.amazonaws.com/123456789012/{queue_name}") you get something like sqs.Queue(url={request_uuid}).

moto might be expecting the full url to the queue ... because the request is made and has a QueueUrl in the params:

api_params = {'QueueUrl': 'XXXXXXXXXXXXXXXX'}.

[nateprewitt]

For everyone else, the issue is recorded for the developers of moto in getmoto/moto#6286 and is currently being looked at. While we wait for a resolution there, the short term solution if you're using moto is going to be to pin both boto3 and botocore.

Downgrading boto3 by itself won't work because every minor version release will always use the latest version of botocore. We wouldn't recommend pinning to boto3<=1.25.

boto3<1.26.127
botocore<1.29.127

[nateprewitt]

@nateprewitt internally AWS url in queues are now request-based and not queue-based.

So instead of sqs.Queue(url="https://sqs.us-west-2.amazonaws.com/123456789012/{queue_name}") you get something like sqs.Queue(url={request_uuid}).

moto might be expecting the full url to the queue ... because the request is made an has a QueueUrl in the params:

api_params = {'QueueUrl': 'XXXXXXXXXXXXXXXX'}.

@Ealameda31 we're aware that what's currently in moto won't function correctly with the new format. A lot of pieces are going to be broken.

What I do want to get a conclusive answer on though is are you seeing functional differences with calls to the production Amazon SQS service (no mocking)? If that's the case, I'd be very interested to get more details on reproduction. If you can open a new ticket if that's the case and we can start tracking it there.

[joaomaranhao]

Yes, this issue doesn't even have to do with moto
After the update of botocore to 1.29.127, create_queue is not returning "QueueUrl" as it should

Could you clarify this a bit more @joaomaranhao? Are you saying live calls to SQS (not using moto) are no longer returning QueueUrl?

I was just saying that the method is not returning "QueueUrl" as documentation says
But actually that call was made with moto! Sorry about that!

[nateprewitt]

I was just saying that the method is not returning "QueueUrl" as documentation says

Sorry I keep asking the same question but want to make sure we're being exact with the issue. Is this with or without moto? If it's with moto, that's entirely expected with the current breakage.

Edit; I saw your edit, thank you!

[Ealameda31]

Sorry, I was trying to clarify that the issue is indeed moto, AWS requests to production are fine.

[kellertk]

Please note: changes to the SQS protocol were reverted in botocore 1.29.129

[nateprewitt]

This issue was remediated for future releases of Boto3 in getmoto/moto#6331. We'll resolve this now that there's a path forward, thanks everyone for your patience.