SendMessage with MessageAttributes on latest version of AWS SDK throws "Attribute MD5 hash mismatch" exception

[jdaigle]

Issue with OpenTelemetry.Instrumentation.AWS

List of all OpenTelemetry NuGet
packages and version that you are
using (e.g. OpenTelemetry 1.3.2):

• OpenTelemetry.Instrumentation.AWS 1.1.0-beta.2
• AWSSDK.SQS 3.7.300.11

Runtime version (e.g. net462, net48, net6.0, net7.0 etc. You can
find this information from the *.csproj file):

• net7.0 and net48

Is this a feature request or a bug?

• Bug

What is the expected behavior?

Messages are sent without exceptions.

What is the actual behavior?

When we use .AddAWSInstrumentation() and then call IAmazonSQS.SendMessageAsync with a message with custom MessageAttributes, an AmazonSQSException exception is thrown with the message "Attribute MD5 hash mismatch".

Additional Context

I think that problem lies in SqsRequestContextHelper.AddAttribute, where the attributes are added to the IRequestContext parameters that is not compatible with the JSON protocol.

In the most recent versions of "AWSSDK.SQS", the SDK changed from the old XML protocol to the new JSON protocol (see https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-sqs-support-json-protocol/).

It would seem the attributes are not properly added to the request, so SQS produces an MD5 hash that does not include the attributes added by OTEL. This doesn't match the MD5 hash generated by the client.

We can workaround by reverting to an older version of "AWSSDK.SQS", but that is not desirable.

[Kielek]

@srprash, @atshaw43, please check.

[birojnayak]

Adding my name and @ppittle to follow up

[ppittle]

Can reproduce issue by slightly modifying TestAWSClientInstrumentation.TestSQSSendMessageSuccessful

    [Fact]
    public void TestSQSSendMessageSuccessful()
    {
        var exportedItems = new List<Activity>();

        var parent = new Activity("parent").Start();

        using (Sdk.CreateTracerProviderBuilder()
                   .AddXRayTraceId()
                   .SetSampler(new AlwaysOnSampler())
                   .AddAWSInstrumentation()
                   .AddInMemoryExporter(exportedItems)
                   .Build())
        {
            var sqs = new AmazonSQSClient(RegionEndpoint.USEast1); // requires .awsconfig with valid creds
            string requestId = @"fakerequ-esti-dfak-ereq-uestidfakere";
            string dummyResponse = "{}";
            //CustomResponses.SetResponse(sqs, dummyResponse, requestId, true); // <-- comment to disable mocking
            var send_msg_req = new SendMessageRequest();
            send_msg_req.QueueUrl = "real-sqs-address";
            send_msg_req.MessageBody = "Hello from OT";
            send_msg_req.MessageAttributes.Add("custom", new MessageAttributeValue { StringValue = "test", DataType = "String"} );
#if NETFRAMEWORK
            sqs.SendMessage(send_msg_req);
#else
            var result = sqs.SendMessageAsync(send_msg_req).Result;
#endif
            Assert.Single(exportedItems);
            Activity awssdk_activity = exportedItems[0];

           // deleted assertions
        }
    }

Test passes with AWSSDK.SQS 3.7.100 and throws exception with AWSSDK.SQS 3.7.300

[ppittle]

NOTE: AWSSDK.SQS 3.7.300 was made minimum in PR #1542

[ppittle]

Investigating root cause and fix.