Amazon.SQS.AmazonSQSException: Submitted policy is over max allowed size

[namado]

Describe the bug

On applications with around 20 SQS subscriptions, the following error occurs randomly when deploying:

Amazon.SQS.AmazonSQSException: Submitted policy is over max allowed size

The deployment works after a few retries.

The policy itself is the one generated in https://github.com/justeat/JustSaying/blob/v6.0.5/JustSaying/AwsTools/MessageHandling/SqsPolicy.cs and there is usually no issues with it.

What we found is that it's submitted every time a new SQS subscription is created and that may be the root cause of the issue on applications with a large amount of subscriptions.

Would it be better to check if the subscription and the policy already exist before creating them, or do you think that is not an issue and the cause is something else?

Steps To reproduce

As this only happens when the queues and subscriptions are already created (and may have messages) so its hard to reproduce.

Expected behaviour

Queues and subscriptions are created with no errors.

Actual behaviour

While queues and subscriptions are created, the following error occurs:

Amazon.SQS.AmazonSQSException: Submitted policy is over max allowed size.    at 
Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionStream(IRequestContext requestContext, IWebResponseData httpErrorResponse, HttpErrorResponseException exception, Stream responseStream)    at 
Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionAsync(IExecutionContext executionContext, HttpErrorResponseException exception)    at 
Amazon.Runtime.Internal.ExceptionHandler`1.HandleAsync(IExecutionContext executionContext, Exception exception)    at 
Amazon.Runtime.Internal.ErrorHandler.ProcessExceptionAsync(IExecutionContext executionContext, Exception exception)    at 
Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)    at 
Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)    at 
Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)    at 
Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)    at 
Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)    at 
Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)    at 
Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)    at 
Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)    at 
Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)    at 
Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)    at 
Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)    at 
Amazon.SimpleNotificationService.AmazonSimpleNotificationServiceClient.SubscribeQueueToTopicsAsync(IList`1 topicArns, ICoreAmazonSQS sqsClient, String sqsQueueUrl)    at 
Amazon.SimpleNotificationService.AmazonSimpleNotificationServiceClient.SubscribeQueueAsync(String topicArn, ICoreAmazonSQS sqsClient, String sqsQueueUrl)    at 
JustSaying.AwsTools.QueueCreation.AmazonQueueCreator.SubscribeQueueAndApplyFilterPolicyAsync(IAmazonSimpleNotificationService amazonSimpleNotificationService, String topicArn, IAmazonSQS amazonSQS, String queueUrl, String filterPolicy) in C:\projects\justsaying\JustSaying\AwsTools\QueueCreation\AmazonQueueCreator.cs:line 81    at 
JustSaying.AwsTools.QueueCreation.AmazonQueueCreator.EnsureTopicExistsWithQueueSubscribedAsync(String region, IMessageSerialisationRegister serialisationRegister, SqsReadConfiguration queueConfig, IMessageSubjectProvider messageSubjectProvider) in C:\projects\justsaying\JustSaying\AwsTools\QueueCreation\AmazonQueueCreator.cs:line 52    at 
JustSaying.JustSayingFluently.TopicHandler[T]() in C:\projects\justsaying\JustSaying\JustSayingFluently.cs:line 297    at 
JustSaying.JustSayingFluently.WithMessageHandler[T](IHandlerResolver handlerResolver) in C:\projects\justsaying\JustSaying\JustSayingFluently.cs:line 267

System information:

• OS: [CentOS 7]
• Library Version [e.g. 6.0.5]
• .NET version - .NetCore 3.1

Additional context

[gkinsman]

Hi there,

Thanks for the report 🙂. Are you able to please give us a little more information on your use case? It sounds like you might be using queue subscription filters, in which case there is a hard limit of 200 per account - are you close to this limit?

I can't think of any reason this would cause transient failures, outside of deleting/recreating policies/queues regularly.

Are you also able to maybe get a hold of a cleansed subscription policy so we can see what's taking up the space?

Thanks!

[namado]

Hi,

We are not using any subscription filters, so that shouldn't be the issue.

We created a console and spammed the queues / notifications creation, even tried doing it when there were pending messages, but were unable to reproduce the issue locally, even though it still happens regularly on production.

The policy generated is:

{
    "Version" : "2012-10-17",
    "Statement" : [
        {
            "Sid" : "sID",
            "Effect" : "Allow",
            "Principal" : {
                "AWS" : "*"
            },
            "Action"    : "sqs:SendMessage",
            "Resource"  : "arn:aws:sqs:eu-west-1:accountID:queueName",
            "Condition" : {
                "ArnLike" : {
                    "aws:SourceArn" : "arn:aws:sns:eu-west-1:accountID:*"
                }
            }
        }
    ]
}

It doesn't look like it fits the exception, but maybe you same some idea what could be the issue here.

This is how we register the queues and notifications

    justSaying
        .WithSqsTopicSubscriber()
        .IntoQueue("myQueue")
            .ConfigureSubscriptionWith(c =>
            {
                c.RetryCountBeforeSendingToErrorQueue = 1;
                c.VisibilityTimeoutSeconds = 300;
            })
        // messages to listen on topics for
        .WithMessageHandler<NotifySubscriber>(handlerResolver)
        .WithMessageHandler<ClientAddressCreated>(handlerResolver)
        .WithMessageHandler<ClientAddressChanged>(handlerResolver)
        .WithMessageHandler<ClientAddressDeleted>(handlerResolver)
        .WithMessageHandler<ClientContactDetailCreated>(handlerResolver)
        .WithMessageHandler<ClientContactDetailChanged>(handlerResolver)
        .WithMessageHandler<ClientContactDetailDeleted>(handlerResolver)
        .WithMessageHandler<ClientChanged>(handlerResolver)
        .WithMessageHandler<ClientCreated>(handlerResolver)
        .WithMessageHandler<DependantCreated>(handlerResolver)
        .WithMessageHandler<DependantChanged>(handlerResolver)
        .WithMessageHandler<DependantDeleted>(handlerResolver)
        .WithMessageHandler<AssetCreated>(handlerResolver)
        .WithMessageHandler<AssetChanged>(handlerResolver)
        .WithMessageHandler<AssetDeleted>(handlerResolver)
        .WithMessageHandler<AssetValuationCreated>(handlerResolver)
        .WithMessageHandler<AssetValuationDeleted>(handlerResolver)
        .WithMessageHandler<IncomeCreated>(handlerResolver)
        .WithMessageHandler<IncomeChanged>(handlerResolver)
        .WithMessageHandler<IncomeDeleted>(handlerResolver)
        .WithMessageHandler<ExpenditureCreated>(handlerResolver)
        .WithMessageHandler<ExpenditureChanged>(handlerResolver)
        .WithMessageHandler<ExpenditureDeleted>(handlerResolver)
        .WithMessageHandler<EmploymentCreated>(handlerResolver)
        .WithMessageHandler<EmploymentChanged>(handlerResolver)
        .WithMessageHandler<EmploymentDeleted>(handlerResolver)
        .WithMessageHandler<IdentityCreated>(handlerResolver)
        .WithMessageHandler<ClientRelationshipCreated>(handlerResolver)
        .WithMessageHandler<ClientRelationshipChanged>(handlerResolver)
        .WithMessageHandler<ClientRelationshipDeleted>(handlerResolver)
        .WithMessageHandler<PlanCreated>(handlerResolver)
        .WithMessageHandler<PlanChanged>(handlerResolver)
        .WithMessageHandler<PlanDeleted>(handlerResolver)
        .WithMessageHandler<PlanValuationCreated>(handlerResolver)
        .WithMessageHandler<PlanValuationDeleted>(handlerResolver)
        .WithSqsPointToPointSubscriber()
        .IntoQueue("myQueue")
        // messages to publish to local queue
        .WithMessageHandler<ChangeOccurred>(handlerResolver)
        .WithSqsMessagePublisher<ChangeOccurred>(config => config.QueueName = "myQueue")
        .WithSqsMessagePublisher<NotifySubscriber>(config => config.QueueName = "myQueue")
        // messages to publish to scheduler queue
        .WithSqsMessagePublisher<ScheduleTimeout>(config => config.QueueName = "myQueue-scheduler")
        .WithSqsMessagePublisher<UnscheduleTimeout>(config => config.QueueName = "myQueue-scheduler");

Thanks for your time.

[jlouros]

Btw, this two issues seem related. Potential regressions from AWS SDK.

• Submitted policy is over max allowed size #211
• Submitted policy is over max allowed size  aws/aws-sdk-net#378