remaining time fluctuations

[AurelienMendes]

Hi !
As already mentionned in the issue #188 by @tim-kos, I observe the same huge estimation time fluctuations.
In my case I use the Dashbord and Tus destination and multiple (4 parallel) uploads
These fluctuations clearly appear when I've many files to uploads which seems to indicates that the specific TUS upload process (POST & PATCH request for one file) is not well estimate.
Or could it be in relation with the wel-known bug #1693 ?

Below some captures to show you ...

...

...

...

...

...

...

...

[goto-bus-stop]

The 0s ones are particularly weird! Have you set a limit: option? From looking at the code, I would expect the 0s to happen sometimes with limit: 1. The upload speed is calculated on a per-file basis and then combined, which I guess doesn't work so well when the files currently being uploaded change constantly.

[AurelienMendes]

Hi @goto-bus-stop ... unfortunately my limit is set to 4

Could it be linked to the bug you already mentionned about the queueing code when we use a limit: N option ? You mentioned it in #1693 ...

Addendum : I've just re-made some tests and it seems that the 0s estimation is when all requests (=limit) are pending simultaneously. Below I set limit: 3 and when the 3 requests are pending the estimation time is 0s

or

In this case that happens very often ...

[AurelienMendes]

Renée, does it sound like something you could understand ?
Can you imagine some others tests that I can do to help you/us to understand the real problem ?

Thanks,
Aurélien

[goto-bus-stop]

I can reproduce it and it's not fixed by the #1736 PR. The cause is that we are using upload speeds of ongoing uploads to estimate the ETA, which is fine for a few big files but is frequently changing if you have many small files. We'll have to do that differently, maybe by tracking the uploaded bytes across all files (ongoing and completed) and averaging it over the last 30 seconds or so.

[AurelienMendes]

Hi Renée ... I don't see any mention about this issue either in Backlog nor in 2.0 & 1.7 tags.
Do you think I'll do better to find a better solution for myself?

[goto-bus-stop]

Yes we won't have time for this very soon most likely. If you have a chance to look into it that would be super appreciated!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[a-kriya]

@goto-bus-stop I'd like to request re-opening this ticket. There's some good discussion here and the issue is well described. I'm noticing same behavior with the AwsS3 plugin.

[a-kriya]

@arturi Is there something in progress or planned that will address this?