♻️ multipart upload chunks are scheduled a few at a time

[GitHK]

What do these changes do?

Schedules less chunks to be uploaded wen running multipart uploads.

• This is also an alternative to ♻️⚗️Add (another) retry for multipart file upload #4483
  I would wait for this to be deployed and see if it improves before trying what is suggested there.

Related issue/s

• improving dynamic-sidecar design osparc-issues#638
• S3TransferError 400 BadRequest with reason RequestTimeout #3531
• fixes When uploading chunks of a file (we gather all the tasks). If one fails then it is added to the bottom of the queue. Ideally it should be put at the beginning of the queue to be picked up as soon as possible. osparc-issues#974

How to test

DevOps Checklist

[codecov]

Codecov Report

Merging #4552 (ff143bf) into master (5f4e812) will increase coverage by 1.4%.
The diff coverage is 84.2%.

@@           Coverage Diff            @@
##           master   #4552     +/-   ##
========================================
+ Coverage    84.3%   85.8%   +1.4%     
========================================
  Files         635     811    +176     
  Lines       30346   36257   +5911     
  Branches      776     532    -244     
========================================
+ Hits        25608   31122   +5514     
- Misses       4554    5007    +453     
+ Partials      184     128     -56     

Flag	Coverage Δ
integrationtests	63.2% <80.4%> (-5.0%)	⬇️
unittests	83.7% <35.0%> (-0.3%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...src/simcore_sdk/node_ports_common/file_io_utils.py	88.4% <82.6%> (+0.3%)	⬆️
packages/service-library/src/servicelib/utils.py	86.7% <90.9%> (+1.5%)	⬆️

... and 497 files with indirect coverage changes

[pcrespov]

A couple of suggestions

[sanderegg]

Cool thanks for this. Also where will you add the overall retry (see DK PR). I think it would also make sense to have it on top.
This one here should allow fail fast, but then the overall retry is also needed

[GitHK]

Cool thanks for this. Also where will you add the overall retry (see DK PR). I think it would also make sense to have it on top.
This one here should allow fail fast, but then the overall retry is also needed

I want this to go to production and see if it makes it better. I want to see if the change has any effect.

Another solution to avoid this problem will be to used rclone, as you have already proposed.

We can wait in adding the retry for a bit. I would still prefer to get away with a "low level" retry and not the retry of the entire file.

[matusdrobuliak66]

👍

[codeclimate]

Code Climate has analyzed commit ff143bf and detected 0 issues on this pull request.

View more on Code Climate.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication