[Feature] Allow custom User-Agents to bypass automation restrictions

[Stridsvagn69420]

The Problem

Out of all websites as of time of writing only the Alphacoders websites block the downloader almost every time. I know that the User-Agent is often the giving factor. The reason why it sets a User-Agent here is because some websites (e.g. Wallhaven) outright refuse to connect/respond normally with agent-less clients.

Experimenting

When trying to download an image from Wallaper Abyss, I get this error message:

wallpaper-dl https://wall.alphacoders.com/big.php?i=1362746

  Fetching https://wall.alphacoders.com/big.php?i=1362746 FAILED HTTP status client error (403 Forbidden) for url (https://wall.alphacoders.com/big.php?i=1362746)

Looking at the response using cURL, there is obviously a block from Cloudflare's side:

curl --head https://wall.alphacoders.com/big.php?i=1362746 -A "wallpaper-dl/0.2.0"

Date: Mon, 17 Jun 2024 20:55:45 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Set-Cookie: [REDACTED]
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Referrer-Policy: no-referrer-when-downgrade
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
CF-Cache-Status: DYNAMIC
Set-Cookie: [REDACTED]
Server: cloudflare
CF-RAY: [REDACTED]
alt-svc: h3=":443"; ma=86400

It also does not matter here if the user agent is set to wallpaper-dl's or none at all, since it's not from a normal browser. But watch what happens when I give it my browser's user agent:

 curl --head https://wall.alphacoders.com/big.php?i=1362746 -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"

HTTP/1.1 200 OK
Date: Mon, 17 Jun 2024 20:59:23 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Set-Cookie: [REDACTED]
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Referrer-Policy: no-referrer-when-downgrade
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
CF-Cache-Status: DYNAMIC
Set-Cookie: [REDACTED]
Server: cloudflare
CF-RAY: [REDACTED]
alt-svc: h3=":443"; ma=86400

The solution

I think the config just needs a little field where we can put a custom User-Agent to bypass this anti-bot detection. Does not work all the time apparently, but it might be useful even as a temporary fix for someone.

[Stridsvagn69420]

I think the config just needs a little field

I removed the config file one version ago, so it will probably be in an environment variable.

[Stridsvagn69420]

Yeah, apparently it does not work...