[Kemono] 429 Too Many Requests #5160
Comments
|
They definitely changed their rate-limiting config (or even activated it in the first place). Just configure sleep and sleep-request https://github.com/mikf/gallery-dl/blob/master/docs/options.md I'm playing around and |
|
sleep-request: Trying 10, 30 seconds now. |
|
I'm actually more hoping for "Hey we hit a rate limit. Wait 30 seconds or longer then try the last request again" vs a relatively static 10-30 second range for Tried to set |
|
@shinji257 @Skaronator It is my understanding that same as bunkr.su they added (or upped/strengthened) their DDosGuard protection that now often will require a captcha to solve before accessing the site or files for download ? cyberdrop-dl has added cookies extraction for this I believe. |
|
Doesn't look like it is that. It looks like they just seem to heavily throttle available connections per client. I get this even during normal browsing from time to time. I think usually if a captcha is hit then you get a 403 error or similar. |
|
This happens with other sites that have rate limiting as well, like desuarchive. Gdl just skips any posts that give a 429. |
|
Yes except it ends up skipping all but maybe the first two artists since it gets the 429 even when trying to pull requests from API. |
|
I'm trying to resume a kemono grab, but gallery-dl grabbing everything up to that point creates the 429 error. Can we get a |
|
Why wouldn't |
|
No idea, I have it set to |
|
Sounds like this setting isn't actually being used. Check with edit: Retries for extractor HTTP requests, i.e. not file downloads, do indeed wait for at least You can pass kemono's |
|
In theory that would work, but there are no "skippable" posts when downloading a discord server... |
|
Discord channels also have an |
|
Interesting, did not know that. Also, it appears just waiting until the next day managed to reset my "too many tries" for now. |
|
So no way to put a pause between downloads? I think just a few seconds would be enough based on my experience with the site. |
|
This has been working so far for me. |
|
The problem I'm having is that even with a decent delay of 10 seconds, it will occasionally skip over one artist due to a 429, and with hundreds of artists and --abort 5, it will potentially skip hundreds of files. In turn I will then have sometimes do a "full" run without --abort which then takes roughly 2 full days with the delays. I'd really just like for it to be able to wait X minutes instead of skipping when it encounters 429, and maybe then I can finally set a delay below 60 seconds... |
|
We can have it retry 429 but there is no way to set a delay on that. I know that (at least) DeviantArt is setup that if a 429 is hit and the active sleep is lower than 30 seconds it adds a second and tries again after that delay. Maybe we can duplicate that over to this to resolve the issue? EDIT: |
|
So it has been running for a day or two. Has hit 429 every now and again but thanks to me adding it as a retry code it seems to be merged into the "5 tries" bit of the application. In the end it manages to get it thanks to the delays I added in. Example logging... (NSFW Warning) |
|
Added a |
* save cookies to tempfile, then rename avoids wiping the cookies file if the disk is full * [deviantart:stash] fix 'index' metadata (mikf#5335) * [deviantart:stash] recognize 'deviantart.com/stash/…' URLs * [gofile] fix extraction * [kemonoparty] add 'revision_count' metadata field (mikf#5334) * [kemonoparty] add 'order-revisions' option (mikf#5334) * Fix imagefap extrcator * [twitter] add 'birdwatch' metadata field (mikf#5317) should probably get a better name, but this is what it's called internally by Twitter * [hiperdex] update URL patterns & fix 'manga' metadata (mikf#5340) * [flickr] add 'contexts' option (mikf#5324) * [tests] show full path for nested values 'user.name' instead of just 'name' when testing for "user": { … , "name": "…", … } * [bluesky] add 'instance' metadata field (mikf#4438) * [vipergirls] add 'like' option (mikf#4166) * [vipergirls] add 'domain' option (mikf#4166) * [gelbooru] detect returned favorites order (mikf#5220) * [gelbooru] add 'date_favorited' metadata field * Update fapello.py get fullsize image instead resized * fapello.py Fullsize image by remove ".md" and ".th" in image url, it will download fullsize of images * [formatter] fix local DST datetime offsets for ':O' 'O' would get the *current* local UTC offset and apply it to all 'datetime' objects it gets applied to. This would result in a wrong offset if the current offset includes DST and the target 'datetime' does not or vice-versa. 'O' now determines the correct local UTC offset while respecting DST for each individual 'datetime'. * [subscribestar] fix 'date' metadata * [idolcomplex] support new pool URLs * [idolcomplex] fix metadata extraction - replace legacy 'id' vales with alphanumeric ones, since the former are no longer available - approximate 'vote_average', since the real value is no longer available - fix 'vote_count' * [bunkr] remove 'description' metadata album descriptions are no longer available on album pages and the previous code erroneously returned just '0' * [deviantart] improve 'index' extraction for stash files (mikf#5335) * [kemonoparty] fix exception for '/revision/' URLs caused by 03a9ce9 * [steamgriddb] raise proper exception for deleted assets * [tests] update extractor results * [pornhub:gif] extract 'viewkey' and 'timestamp' metadata (mikf#4463) mikf#4463 (comment) * [tests] use 'datetime.timezone.utc' instead of 'datetime.UTC' 'datetime.UTC' was added in Python 3.11 and is not defined in older versions. * [gelbooru] add 'order-posts' option for favorites (mikf#5220) * [deviantart] handle CloudFront blocks in general (mikf#5363) This was already done for non-OAuth requests (mikf#655) but CF is now blocking OAuth API requests as well. * release version 1.26.9 * [kemonoparty] fix KeyError for empty files (mikf#5368) * [twitter] fix pattern for single tweet (mikf#5371) - Add optional slash - Update tests to include some non-standard tweet URLs * [kemonoparty:favorite] support 'sort' and 'order' query params (mikf#5375) * [kemonoparty] add 'announcements' option (mikf#5262) mikf#5262 (comment) * [wikimedia] suppress exception for entries without 'imageinfo' (mikf#5384) * [docs] update defaults of 'sleep-request', 'browser', 'tls12' * [docs] complete Authentication info in supportedsites.md * [twitter] prevent crash when extracting 'birdwatch' metadata (mikf#5403) * [workflows] build complete docs Pages only on gdl-org/docs deploy only docs/oauth-redirect.html on mikf.github.io/gallery-dl * [docs] document 'actions' (mikf#4543) or at least attempt to * store 'match' and 'groups' in Extractor objects * [foolfuuka] improve 'board' pattern & support pages (mikf#5408) * [reddit] support comment embeds (mikf#5366) * [build] add minimal pyproject.toml * [build] generate sdist and wheel packages using 'build' module * [build] include only the latest CHANGELOG entries The CHANGELOG is now at a size where it takes up roughly 50kB or 10% of an sdist or wheel package. * [oauth] use Extractor.request() for HTTP requests (mikf#5433) Enables using proxies and general network options. * [kemonoparty] fix crash on posts with missing datetime info (mikf#5422) * restore LD_LIBRARY_PATH for PyInstaller builds (mikf#5421) * remove 'contextlib' imports * [pp:ugoira] log errors for general exceptions * [twitter] match '/photo/' Tweet URLs (mikf#5443) fixes regression introduced in 40c0553 * [pp:mtime] do not overwrite '_mtime' for None values (mikf#5439) * [wikimedia] fix exception for files with empty 'metadata' * [wikimedia] support wiki.gg wikis * [pixiv:novel] add 'covers' option (mikf#5373) * [tapas] add 'creator' extractor (mikf#5306) * [twitter] implement 'relogin' option (mikf#5445) * [docs] update docs/configuration links (mikf#5059, mikf#5369, mikf#5423) * [docs] replace AnchorJS with custom script use it in rendered .rst documents as well as in .md ones * [text] catch general Exceptions * compute tempfile path only once * Add warnings flag This commit adds a warnings flag It can be combined with -q / --quiet to display warnings. The intent is to provide a silent option that still surfaces warning and error messages so that they are visible in logs. * re-order verbose and warning options * [gelbooru] improve pagination logic for meta tags (mikf#5478) similar to 494acab * [common] add Extractor.input() method * [twitter] improve username & password login procedure (mikf#5445) - handle more subtasks - support 2FA - support email verification codes * [common] update Extractor.wait() message format * [common] simplify 'status_code' check in Extractor.request() * [common] add 'sleep-429' option (mikf#5160) * [common] fix NameError in Extractor.request() … when accessing 'code' after an requests exception was raised. Caused by the changes in 566472f * [common] show full URL in Extractor.request() error messages * [hotleak] download files with 404 status code (mikf#5395) * [pixiv] change 'sanity_level' debug message to a warning (mikf#5180) * [twitter] handle missing 'expanded_url' fields (mikf#5463, mikf#5490) * [tests] allow filtering extractor result tests by URL or comment python test_results.py twitter:+/i/web/ python test_results.py twitter:~twitpic * [exhentai] detect CAPTCHAs during login (mikf#5492) * [output] extend 'output.colors' (mikf#2566) allow specifying ANSI colors for all loglevels (debug, info, warning, error) * [output] enable colors by default * add '--no-colors' command-line option --------- Co-authored-by: Luc Ritchie <luc.ritchie@gmail.com> Co-authored-by: Mike Fährmann <mike_faehrmann@web.de> Co-authored-by: Herp <asdf@qwer.com> Co-authored-by: wankio <31354933+wankio@users.noreply.github.com> Co-authored-by: fireattack <human.peng@gmail.com> Co-authored-by: Aidan Harris <me@aidanharr.is>
After a short period of time I get a 429 error. It lasts for a bit then goes away (I'd say 30 seconds?) however I was hoping someone had a way to mitigate the problem in the configuration so I can actually do a run without hitting this?
As of right now when 429 is hit for Kemono it doesn't even slow down. It just continues to spam skipping over artists until it gets to the end of the list.
The text was updated successfully, but these errors were encountered: