feat(job_attachments): enhance handling S3 timeout errors and BotoCoreError

[gahyusuh]

Improve error handling for S3 requests by

• adding "retries" configuration to the S3 client
• adding BotoCoreError handling to cover S3 timeout errors (e.g., ReadTimeoutError, ConnectTimeoutError)

Although not related to the main changes here, made an additional improvement:
Update the script upload_cancel_test.py to create large files in smaller sized chunks to prevent MemoryError.

What was the problem/requirement? (What/Why)

There was an issue where a long and non-user-friendly error stack trace was printed to the console during file uploads when trying to submit a job over a slow internet bandwidth. (it was confirmed that they could GET and DELETE objects in their bucket with permissions, so it seems purely a bandwidth-related issue.) The reported timeout error was ReadTimeoutError, a botocore exception. While we are handling ClientError from botocore, we lacked handling for other errors like ReadTimeoutError.

What was the solution? (How)

1. Use standard retry mode: include a "retries", setting the mode to "standard", in the S3 client's configuration. This is an more advanced retry logic than the default retry mode called legacy.
2. Catch BotoCoreError: This botocore exception is a base class for many errors related to boto3 calls, which typically occur during the preparation phase of AWS service calls due to issues like network connection problems, or authentication failures. I have added more user-friendly error messages to better communicate these issues to users.

What is the impact of this change?

More robust error handling for S3 requests, providing clear and non-cumbersome error messages to users when S3 requests failed with BotoCoreError.

How was this change tested?

• Unit tests: added a couple new unit tests covering the new error handling path.
• Ran Integ tests and made sure all passed.
• Manually reproduced the ReadTimeoutError while submitting a job to verify that the new error message is displayed as expected:

Submitting to Queue: Queue02
Job submission contains 1 files totaling 10.74 GB.  All files will be uploaded to S3 if they are not already present in the job attachments bucket.

Do you wish to proceed? [Y/n]: 
Hashing Attachments  [####################################]  100%
Hashing Summary:
    Processed 0 files totaling 0.0 B.
    Skipped re-processing 1 files totaling 10.74 GB.
    Total processing time of 0.0038 seconds at 0.0 B/s.

Failed to upload job attachments:
An issue occurred with AWS service request while checking for the existence of an object in the S3 bucket: Read timeout on endpoint URL: "https://s3.us-west-2.amazonaws.com/job-attachments-bucket/rootPrefix/Data/818cb10ad378806591de713e78b9b1f9.xxh128"
This could be due to temporary issues with AWS, internet connection, or your AWS credentials. Please verify your credentials an network connection. If the problem persists, try again later or contact support for further assistance.

Was this change documented?

No.

Is this a breaking change?

No.

[marofke]

Great work here recreating the error and making a much better UX from timeouts!

LGTM! 🚢 🦈