S3 TransferUtility: Lower the minimum notification threshold of 128KB

[hipwelljo]

Describe the bug
I am utilizing the TransferUtility to upload a file and using TransferObserver to set a TransferListener to be informed of state changes, progress changes, and any errors should one occur. The file is uploading successfully, however I am not being informed of any upload progress aside from 0% and 100%. (The callback is often triggered multiple times for the same percentage which is already being tracked in issue #406.) As you can see here, I am getting no progress aside from started/finished.

12-12 14:04:21.080 10585-10585/com.x.x D/RestHelper: Upload state changed to IN_PROGRESS
12-12 14:04:21.080 10585-10585/com.x.x D/RestHelper: ID:9 bytesCurrent: 0 bytesTotal: 35980 0%
12-12 14:04:30.507 10585-10585/com.x.x D/RestHelper: ID:9 bytesCurrent: 35980 bytesTotal: 35980 100%
12-12 14:04:30.523 10585-10585/com.x.x D/RestHelper: ID:9 bytesCurrent: 35980 bytesTotal: 35980 100%
12-12 14:04:30.523 10585-10585/com.x.x D/RestHelper: ID:9 bytesCurrent: 35980 bytesTotal: 35980 100%
12-12 14:04:30.523 10585-10585/com.x.x D/RestHelper: Upload state changed to COMPLETED

To Reproduce
Steps to reproduce the behavior:

1. Attempt to upload a file and log the upload progress (while on a slow connection so it takes some time)

Which AWS service(s) are affected?
AWS S3 TransferUtility

Expected behavior
Progress should be reported as bytes are uploaded

Screenshots
None

Environment(please complete the following information):

• SDK Version: 2.9.0

Device Information (please complete the following information):

• Device: Emulator - Pixel API 22
• Android Version: 5.1.1
• Specific to simulators: Have not tried physical device yet

Code

        TransferUtility transferUtility =
                TransferUtility.builder()
                        .context(context.getApplicationContext())
                        .awsConfiguration(AWSMobileClient.getInstance().getConfiguration())
                        .s3Client(new AmazonS3Client(AWSMobileClient.getInstance()))
                        .build();

        final String filename = "image_" + randomAlphaNumeric(10) + ".jpg";

        File file = scaleImage(uri, context.getCacheDir().getPath() + File.separator + filename);

        TransferObserver transferObserver = transferUtility.upload("bucketname", filename, file);
        transferObserver.setTransferListener(new TransferListener() {
            @Override
            public void onStateChanged(int id, TransferState state) {
                Log.d(TAG, "Upload state changed to " + state);
            }

            @Override
            public void onProgressChanged(int id, long bytesCurrent, long bytesTotal) {
                float percentDonef = ((float) bytesCurrent / (float) bytesTotal) * 100;
                int percentDone = (int)percentDonef;

                Log.d(TAG, "ID:" + id + " bytesCurrent: " + bytesCurrent
                        + " bytesTotal: " + bytesTotal + " " + percentDone + "%");
            }

            @Override
            public void onError(int id, Exception ex) {
                Log.e(TAG, ex);
            }
        });

If helpful, this is how I initialize it in my startup activity:

AWSMobileClient.getInstance().initialize(getApplicationContext(), new Callback<UserStateDetails>() {
            @Override
            public void onResult(UserStateDetails result) {
                Log.d(TAG, "AWSMobileClient initialized. User State is " + result.getUserState());

                TransferUtility transferUtility =
                        TransferUtility.builder()
                                .context(getApplicationContext())
                                .awsConfiguration(AWSMobileClient.getInstance().getConfiguration())
                                .s3Client(new AmazonS3Client(AWSMobileClient.getInstance()))
                                .build();
                transferUtility.resumeAllWithType(UPLOAD); //resume any uploads that were paused on app termination
            }

            @Override
            public void onError(Exception e) {
                Log.e(TAG, "Initialization error.", e);
            }
        });

        getApplicationContext().startService(new Intent(getApplicationContext(), TransferService.class));

(As an aside, is there a way to get the already initialized transfer utility later when I go to upload? Is it creating two different transferUtilitys as it is now?)

Other Info
I have also tried configuring the client to change the notification threshold to see if that would cause more progress to be reported but I still only see 0% and 100%.

AmazonS3Client client = new AmazonS3Client(AWSMobileClient.getInstance());
client.setNotificationThreshold(10);

To simulate slow network conditions, I enabled the network link conditioner on macOS to simulate an edge data connection, so the file doesn't upload quickly.

[sunchunqiang]

Hi @hipwelljo

Thanks your reporting. I have a question for the issue.
How large are the files you uploaded?

[hipwelljo]

They are photos taken with the device’s camera. In the emulator this is the funky animating box they play as a camera feed. So a few MB at max, usually about or under 1 MB I’d say.

[hipwelljo]

I am now trying this on a physical device (Google pixel android 7.1.2) and I see the same behavior with about a 0.15 MB photo - only 0 and 100%. Just now I did see an 89% reported. This is on an edge network connection so it takes a bit of time to upload. I tried a 1.6 MB photo and it did report more progress. Is there a way to reliably get progress updates if the file is small yet the internet connection isn't great?

[sunchunqiang]

This appears to be a limitation in our code that doesn't allow setting the granularity very well. We will investigate enabling finer granularity progress reporting and update this thread.

[mutablealligator]

@hipwelljo Just wanted to know if this issue is still persistent after customizing the notification threshold?

[hipwelljo]

Yes, I’ve set it to 10 currently.

[mutablealligator]

@hipwelljo The minimum notification threshold that AmazonS3Client supports in 128KB. This is because in our HTTP Client in Core Runtime of the SDK, the buffer size is set to 128KB. That's the reason why you did not get an update for files of size 100KB.

[mutablealligator]

Besides that, 2.11.1 version of the SDK contains improved exception handling and pause/resume handling and fixes the progress reporting issue when a transfer is paused and resumed.

Root cause analysis:

1. When a transfer is paused by the user or due to network drop, the thread that is managing the multipart transfer is interrupted. When the thread is interrupted and encounters an exception, there is a case where we skip setting the transfer to PAUSED or WAITING_FOR_NETWORK, and instead moves to FAILED.

2. When a transfer is paused by the user or due to network drop, the thread that is managing the multipart transfer is interrupted. When a multi-part transfer (N parts) is interrupted, there are X number of parts that may have been completed, Y number of parts that may been in progress and Z number of parts that are yet to be started (X + Y + Z = N). When a part that is in progress is interrupted, and then resumed at a later point, the part is retried from the beginning, however the progress made in the previous attempt that is interrupted is not reset from the total progress. This caused the progress reported to go over 100%.

Description of the fix:

The fix involves

1. improving the exception handling of transfers:
   Improved the state, progress and error reporting when the transfers are interrupted.

• When the transfer is paused or cancelled by the user, the state is reported correctly.
• When the transfer is interrupted because of a network drop, the state is set to WAITING_FOR_NETWORK when the TransferNetworkLossHandler is used.
• When the transfer is interrupted otherwise, the transfer is set to FAILED and the exception is reported via TransferListener.onError callback.

2. Fix the progress reporting to not go beyond 100% and reset when a part failed.

See 2.11.1 for more information.

[hipwelljo]

Gotcha. Is there a way to reduce that buffer size? It seems our photos are generally within the range of 50-100 KB.

[mutablealligator]

@hipwelljo Unfortunately, the current minimum that the SDK supports in 128KB.

[hipwelljo]

Ok! Then this can be marked enhancement request instead of bug ;)

[mutablealligator]

@hipwelljo I have made the necessary modifications to the title and the state of the issue. Thank you!

[gpanshu]

Checking in here as this issue is now more than 2 years old and Android devices do not take pictures anymore that are lower than 128KB. I am going to go ahead and close this issue as a "Wont do" based on what Kvasukib mentioned and the limitations. Feel free to reopen if this still persists.