[Danbooru] Cloudflare challenge warning + 403 error

[MarqFJA87]

Despite using the username and API key, trying to download from Danbooru - or even attempt to fetch the list of keywords applicable to the site - runs into a Cloudflare challenge warning that's followed with a 403 error.

Here's the verbose log.

C:\Users\marq_>gallery-dl -K https://danbooru.donmai.us/posts/6074569 --verbose
[gallery-dl][debug] Version 1.24.5
[gallery-dl][debug] Python 3.10.0 - Windows-10-10.0.19045-SP0
[gallery-dl][debug] requests 2.26.0 - urllib3 1.26.7
[gallery-dl][debug] Configuration Files ['%USERPROFILE%\\gallery-dl\\config.json']
[gallery-dl][debug] Starting KeywordJob for 'https://danbooru.donmai.us/posts/6074569'
[danbooru][debug] Loading cookies from 'C:/Users/marq_/gallery-dl/cookies.txt'
[danbooru][debug] Using HTTP Basic Auth for user 'MarqFJA87'
[danbooru][debug] Using DanbooruPostExtractor for 'https://danbooru.donmai.us/posts/6074569'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): danbooru.donmai.us:443
[urllib3.connectionpool][debug] https://danbooru.donmai.us:443 "GET /posts/6074569.json HTTP/1.1" 403 None
[danbooru][warning] Cloudflare challenge
[danbooru][error] HttpError: '403 Forbidden' for 'https://danbooru.donmai.us/posts/6074569.json'

[mikf]

https://danbooru.donmai.us/forum_topics/22822
https://danbooru.donmai.us/forum_topics/22987
https://danbooru.donmai.us/forum_topics/23275

#3128

Try using a different --user-agent than the default Firefox 102.0, for example gdl/1.24.5.

[MarqFJA87]

It worked. I assume that I should just use the user agent extractor option to make Danbooru downloads do this automatically from now on?

[RarogCmex]

Maybe user-agent should be randomized by default?

[AdjointOperator]

The issue seems to be with the CF browser integrity check or so. Using modern browsers with unmatching system UA will result in 403. However, ancient UAs (e.g., ones from IE6) seem to work without any issues...

[marlekal]

It worked. I assume that I should just use the user agent extractor option to make Danbooru downloads do this automatically from now on?

how do you do that please? i wish to make it automatic since i have mutiple links from many platforms and singeling out danbooru is gonna take me a long time
unless this is gonna get patched, then i can wait

[MarqFJA87]

It worked. I assume that I should just use the user agent extractor option to make Danbooru downloads do this automatically from now on?

how do you do that please? i wish to make it automatic since i have mutiple links from many platforms and singeling out danbooru is gonna take me a long time unless this is gonna get patched, then i can wait

It's in the list of configuration options here. Just add this line to the Danbooru extractor:

"user-agent": "gdl/1.24.5"

Make sure you add a comma at the end if it's not the only/final option within the extractor.

[rautamiekka]

Need to remember that some Danbooru sites, such as e621/e962 (although it's a heavily modified codebase), may have limits on user agents. Case in point:

User Agents
A non-empty User-Agent header is required for all requests. Please pick a descriptive User-Agent for your project.
You are encouraged to include your e621 username, so that you may be contacted if your project causes problems.
For example: MyProject/1.0 (by username on e621)

Important!
Do not impersonate a browser user agent, as this will get you blocked.

Due to frequent abuse, default user agents for programming languages and libraries are usually blocked.
Please make sure that you are defining a user agent that is in line with our policy.

In some cases, you may be unable to set a custom header for your requests.
This may be because you are creating a userscript, a browser extension, or otherwise something that works within a browser.
If that is the case, please attach an additional url quest parameter named _client, and set it to your user-agent.

In such cases the appropriate extractor needs to either forcibly override the user-defined one or limit the allowed ones. I vote for the former cuz coming up with the specifications doesn't sound like worth it.

[mikf]

gallery-dl now automatically sends gallery-dl/<version> as UA for all danbooru instances.

---

In such cases the appropriate extractor needs to either forcibly override the user-defined one or limit the allowed ones. I vote for the former cuz coming up with the specifications doesn't sound like worth it.

e621 always automatically sent gallery-dl/<version> (by mikf) as UA. It also enforces a 1s wait time between API requests.