Add a retry logic with backoff and jitter

[karan6181]

Description of changes:

• Add a retry logic with backoff and jitter to upload and download functionality
• Enqueue the index.json upload after all the shard uploads has been finished.
• Kill the main thread if one of the upload threads receives Exception.

Issue #, if available:

STR-136
#451

Testing

Integration testing

- UC Volume

• Verified that I am able to see the REQUEST_LIMIT_EXCEEDED exception by spinning approx. 500 processes where it hits the 400 request limit per secs.
• Having more than one attempts doesn't crash the job since it retries downloading the shard file again.
• Verified FileNotFound Exception from UC Volume by providing a corrupt path.

- Other cloud providers

• Tested invalid path with S3 and OCI and able to see the relevant FileNotFound Exception.
• Ran single node model training on S3 and OCI with correct dataset path and I don't see an Exception.

Merge Checklist:

Put an x without space in the boxes that apply. If you are unsure about any checklist, please don't hesitate to ask. We are here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

• I have read the contributor guidelines
• This is a documentation change or typo fix. If so, skip the rest of this checklist.
• I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the MosaicML team.
• I have updated any necessary documentation, including README and API docs (if appropriate).

Tests

• I ran pre-commit on my change. (check out the pre-commit section of prerequisites)
• I have added tests that prove my fix is effective or that my feature works (if appropriate).
• I ran the tests locally to make sure it pass. (check out testing)
• I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes.

[snarayan21]

Some minor comments, other than that lgtm. Using the package might be easier down the road but this also works

[knighton]

Noice