S3 Parallel Object Uploads Eventually Cause Exception: 'The response ended prematurely.'

[kakins]

Here's my scenario:

• Writing a little test harness that uploads about 200 1MB Objects to S3 every 5 seconds
• Using PutObjectAsync
• Each upload operation breaks the 200 Objects into batches of 10 -- so about 20 upload operations in parallel
• I'm fairly certain each upload of 200 doesn't fully complete in under 5 seconds, and the next upload of 200 begins
• This works fine for about 10-15 seconds, then I start getting this exception

Questions:

Everything I've read so far says this is caused by the server itself. So AWS for some reason is closing the connection? If so, does anyone know why?

Could this possibly have anything to do with the SDK?

[ashishdhingra]

@kakins Good morning. Could you please share the sample code and more detailed logs to troubleshoot the issue? You might also want to enable verbose logging using the below code:

Amazon.AWSConfigs.LoggingConfig.LogResponses = Amazon.ResponseLoggingOption.Always;
Amazon.AWSConfigs.LoggingConfig.LogTo = Amazon.LoggingOptions.SystemDiagnostics;
Amazon.AWSConfigs.AddTraceListener("Amazon", new System.Diagnostics.ConsoleTraceListener());

Thanks,
Ashish

[kakins]

Ah, yes didn't realize this was an option. Let me try this and I'll get back with you thanks!

[kakins]

@ashishdhingra Ok, so I can't really provide any more detailed logs, because this is all I get, even after enabling tracing like you suggested.

Once I reach uploading about 200 Objects uploading every 5 seconds is when I start to notice the HTTP and System.IO exceptions. I do notice that the next set of 200 Objects begin uploading before the previous set completes. So I'm wondering if I'm exhausting something internally in the SDK -- but the error shown in my original post seems to indicate that the server ended the request.

Here's some minimal sample code. I've toyed around a lot with async here while trying to batch requests in different ways.

public class TestRunner
{
    private readonly S3Service _s3Service;
    
    public TestRunner(S3Service s3Service)
    {
        _s3Service = s3Service;
    }

    public async Task Upload()
    {
        // Sample data here...
        // Sorry I didn't create a method to generate exactly what I'm sending, but each JSON object is about 1MB
        var data = Enumerable.Range(200, 200).Select(idx => "{}");
        var tasks = new List<Task>();
        var stopwatch = new Stopwatch();

        stopwatch.Start();
        while (stopwatch.Elapsed < TimeSpan.FromSeconds(30))
        {
            tasks.Add(UploadObjectAsync(data, "my-bucket"));
            await Task.Delay(TimeSpan.FromSeconds(5));
        }
        stopwatch.Stop();

        await Task.WhenAll(tasks);
    }

    private async Task UploadObjectAsync(IEnumerable<string> data, string bucketName)
    {
        // Attempting to batch uploads in chunks of 10
        var tasks = data.Chunk(10).Select(async batch =>
        {
            await _s3Service.UploadObjectsAsync(batch, bucketName);
        });

        await Task.WhenAll(tasks);
    }
}

public class S3Service
{
    private readonly IAmazonS3 _s3Client;

    public S3Service(IAmazonS3 s3Client)
    {
        _s3Client = s3Client;
    }

    public async Task UploadObjectsAsync(IEnumerable<string> objects, string bucketName)
    {
        var uploadTasks = objects.Select(contentBody => _s3Client.PutObjectAsync(CreatePutObjectRequest(contentBody, bucketName)));
        var responses = await Task.WhenAll(uploadTasks);
    }

    private PutObjectRequest CreatePutObjectRequest(string contentBody, string bucketName)
    {
        return new PutObjectRequest
        {
            ContentBody = contentBody,
            BucketName = bucketName,
            Key = $"{Guid.NewGuid()}",
        };
    }
}

[Johnjon987]

[kakins]

?

[kakins]

@ashishdhingra Any thoughts or follow up on my previous comment? Any further info I can provide?

[kakins]

Still waiting on a reply....

[ashishdhingra]

@kakins Apologies for the delayed reply. Somehow, the issue is not reproducible using the code you shared. Attaching it below for reference:

using Amazon.S3;
using Amazon.S3.Model;
using System.Diagnostics;

Amazon.AWSConfigs.LoggingConfig.LogResponses = Amazon.ResponseLoggingOption.Always;
Amazon.AWSConfigs.LoggingConfig.LogTo = Amazon.LoggingOptions.SystemDiagnostics;
Amazon.AWSConfigs.AddTraceListener("Amazon", new System.Diagnostics.ConsoleTraceListener());

IAmazonS3 _s3client = new AmazonS3Client();
var testRunner = new TestRunner(new S3Service(_s3client));
await testRunner.Upload();

Console.WriteLine("FINISHED");
public class TestRunner
{
    private readonly S3Service _s3Service;

    public TestRunner(S3Service s3Service)
    {
        _s3Service = s3Service;
    }

    public async Task Upload()
    {
        // Sample data here...
        // Sorry I didn't create a method to generate exactly what I'm sending, but each JSON object is about 1MB
        var data = Enumerable.Range(200, 200).Select(idx => "{}");
        var tasks = new List<Task>();
        var stopwatch = new Stopwatch();

        stopwatch.Start();
        while (stopwatch.Elapsed < TimeSpan.FromSeconds(30))
        {
            tasks.Add(UploadObjectAsync(data, "my-bucket"));
            await Task.Delay(TimeSpan.FromSeconds(5));
        }
        stopwatch.Stop();

        await Task.WhenAll(tasks);
    }

    private async Task UploadObjectAsync(IEnumerable<string> data, string bucketName)
    {
        // Attempting to batch uploads in chunks of 10
        var tasks = data.Chunk(10).Select(async batch =>
        {
            await _s3Service.UploadObjectsAsync(batch, bucketName);
        });

        await Task.WhenAll(tasks);
    }
}

public class S3Service
{
    private readonly IAmazonS3 _s3Client;

    public S3Service(IAmazonS3 s3Client)
    {
        _s3Client = s3Client;
    }

    public async Task UploadObjectsAsync(IEnumerable<string> objects, string bucketName)
    {
        var uploadTasks = objects.Select(contentBody => _s3Client.PutObjectAsync(CreatePutObjectRequest(contentBody, bucketName)));
        var responses = await Task.WhenAll(uploadTasks);
    }

    private PutObjectRequest CreatePutObjectRequest(string contentBody, string bucketName)
    {
        return new PutObjectRequest
        {
            ContentBody = contentBody,
            BucketName = bucketName,
            Key = $"{Guid.NewGuid()}",
        };
    }
}

I'm unsure if this is an intermittent or throttling issue at your end.

Thanks,
Ashish

@github-actions proposed-answer

[kakins]

I will continue to investigate. We have another effort to conduct some performance tests distributing the uploads. We assumed that our issue may be with the amount of data uploaded simultaneously from a single client. For instance, one client uploading several MBs/sec concurrently.

It does seems strange though that the response we received was that the server (AWS) ended the request, which does not seem to reflect an internal error on our end or with our HTTP library.

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.