Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timeout exception when downloading large file #47

Closed
Lqlsoftware opened this issue Dec 10, 2020 · 3 comments · Fixed by #50
Closed

Timeout exception when downloading large file #47

Lqlsoftware opened this issue Dec 10, 2020 · 3 comments · Fixed by #50
Labels
bug Something isn't working

Comments

@Lqlsoftware
Copy link
Contributor

Hi!
I am very interested in this project, and your code is so clean and readable to me. I would like to do something for this repo (or others XD).
There are some exceptions when using the script downloading files from channels:
When downloading large size of files, there is a great chance that some small-sized images will be downloaded together.
The await-sync programming model will stall other threads and coming to a timeout eventually, the last_message_id was not flush into the config.yaml before the timeout exception (damn...).
I'm trying to edit the script to prevent timeout exception when downloading a large file, and save the last_message_id when the exception occurs.
Do you have ideas to make the script better?

@Lqlsoftware
Copy link
Contributor Author

My bad. This issue is similar to #43. But it seems that I found the reason which causes the exception

@Dineshkarthik Dineshkarthik added the bug Something isn't working label Dec 10, 2020
@Dineshkarthik
Copy link
Owner

@Lqlsoftware glad, you liked this project and found it useful. Thanks for getting into the details of the issue.
Yeah, in regards to the last_message_id if you are going to create a patch I would suggest changing the logic of the last_message_id to update in config.yaml for each batch of messages that are downloaded so even if the script is manually interrupted it would have the latest message_id.

We can also work on fixing the asyncio timeout issue by tinkering and changing how the coroutine & tasks are created currently - media_downloader.py#L175

@Lqlsoftware
Copy link
Contributor Author

Sure, I will pull a request for updating last_message_id in config.yaml later this day.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants