Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix uukanshu possible failure on synopsis #2279

Merged
merged 1 commit into from
Feb 25, 2024
Merged

Conversation

camp00000
Copy link

@camp00000 camp00000 commented Feb 23, 2024

Found on #2270

This source suffers from rate-limiting after a certain point (only X amount of requests per hour/day allowed, etc) with cloudflare or similar blocking any further requests after a certain point for some time -> is there any way to force proxies or make a popup appear on all log levels for the user to let them know of this limitation that can/should be bypass with proxies if it occurs?

@dipu-bd dipu-bd merged commit 8288270 into dipu-bd:dev Feb 25, 2024
4 checks passed
@dipu-bd
Copy link
Owner

dipu-bd commented Feb 25, 2024

There is configuration for ratelimiting here: self.init_executor(ratelimit=0.5)

Check https://github.com/dipu-bd/lightnovel-crawler/blob/dev/lncrawl/core/taskman.py#L56

@dipu-bd
Copy link
Owner

dipu-bd commented Feb 25, 2024

I think making a message appearing to the user everytime they try to download a novel can become annyoing to some. However, we can add a section in the REAMDE to describe usage of proxies and ratelimiting.

@camp00000
Copy link
Author

camp00000 commented Feb 25, 2024

Thanks for the info on the rate_limiting, I realized I may have confused this source's rate-limiting with that of another since I can't find the comments/issues I had in mind right now, but regardless I'll keep this in mind going forward and maybe will create a PR updating the README at some point in the future.

Update: I confused the rate-limiting of this source with that of 69xinshu, see #2269 for more details

@camp00000 camp00000 deleted the uukanshu branch March 10, 2024 12:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants