Grouping media files based on file size for downloading

[Dineshkarthik]

Currently, the media files are concurrently downloaded on the basis of the order of the messages.
This leads to some async/await timeout errors when a large media file ends up getting loaded along with small files which get downloaded in a few seconds while the large file takes a couple of minutes as reported by @Lqlsoftware in #47

@Lqlsoftware also pushed a fix which changed the strategy of how the last_messaged_id is updated in the config file for every batch of media files downloaded concurrently thus the progress is saved when the above-mentioned timeout error occurs - #50

The idea proposed is to group the messages/media files to download based on the size of the files which has the following benefits:

• Improves performance as the latency between asyncio coroutines completion waiting for other coroutines in a specific message batch to complete is reduced as files which are similar in size are downloaded concurrently.
• This fixes the asyncio coroutine timeout issue There is an error in the file in the download channel, the log file is here #43

Please share your suggestion regarding the above idea.

UPDATE:

The above-mentioned timeout error occurs due to FILE_REFERENCE_EXPIRED exception as mentioned here and in issue #52 and is fixed here #63

[Lqlsoftware]

Agreed.
My current solution for downloading multiple sizes of files is to set pagination_limit small enough to avoid timeout (like 4).
I think grouping it by size will fix this issue, but there are somethings we need to pay attention to:

• The larger-sized files may take several minutes to download while small-sized files already synced. It is hard to determine the current last_messaged_id.
• Base on the first point, there needs to be some mechanism to avoid duplicate downloading when the next execution of the script comes.

Or can we get all the list of files to be downloaded, so that we can remove last_messaged_id? The messages iter API in Program seems to be a generator implemented by yield.

[Dineshkarthik]

• Reducing the pagination_limit to a smaller number like 4 will have a huge impact on the concurrency of the download. We will end up increasing script execution time which I reduced by introducing this concurrency using asyncio - Parallel downloads #8
• The last_message_id is used to maintain the state of message/media files that are downloaded during the last run, and pick up from that offset on the next run thus last_message_id can't be removed.