Access denied for some user agents

[ibnesayeed]

Find an example "in the wild" where the access to the resource is denied by default when using cURL, but not when specifying a different user agent (e.g., -A "Mozilla").

[calebkbrad]

Amazon.com will reject a request with a 503 code when using a curl request with no specified user agent as shown below

curl -IL https://amazon.com
HTTP/1.1 301 Moved Permanently
Server: Server
Date: Fri, 28 Oct 2022 21:15:24 GMT
Content-Type: text/html
Content-Length: 163
Connection: keep-alive
Location: https://www.amazon.com/

HTTP/2 503
server: Server
date: Fri, 28 Oct 2022 21:15:24 GMT
content-type: text/html
strict-transport-security: max-age=47474747; includeSubDomains; preload
x-amz-rid: 6ZF59FKT3BTJMHPKZB99
vary: Content-Type,Accept-Encoding,User-Agent
last-modified: Tue, 25 Oct 2022 19:37:40 GMT
etag: "a6f-5ebe10a81c900"
accept-ranges: bytes
x-served-by: cache-iad-kiad7000020-IAD

However, adding the -A flag with "Mozilla" as a user agent will result in a 200.

curl -IL -A "Mozilla" https://amazon.com
HTTP/1.1 301 Moved Permanently
Server: Server
Date: Fri, 28 Oct 2022 21:15:13 GMT
Content-Type: text/html
Content-Length: 163
Connection: keep-alive
Location: https://www.amazon.com/

HTTP/2 200
server: Server
date: Fri, 28 Oct 2022 21:15:13 GMT
content-type: text/html
strict-transport-security: max-age=47474747; includeSubDomains; preload
x-amz-rid: 5XA1KJ77Y5XBXZWBCQBJ
vary: Content-Type,Accept-Encoding,User-Agent
x-served-by: cache-iad-kcgs7200151-IAD

[Badjedi04]

geekforgeeks.com will deny the request with 403 status code when using a curl request with no specified user agent as shown below
' ' ' '
$ curl -ILs https://www.geekforgeeks.com
HTTP/1.1 302 Found
Date: Sat, 29 Oct 2022 01:30:38 GMT
Server: Apache/2.4.38 (Debian)
Set-Cookie: __tad=1667007038.5205528; expires=Tue, 26-Oct-2032 01:30:38 GMT; Max-Age=315360000
Location: http://ww38.geekforgeeks.com/?subid1=20221029-1230-3879-b78e-8b36da1f9e97
Connection: close
Content-Type: text/html; charset=UTF-8

HTTP/1.1 403 Forbidden
Date: Sat, 29 Oct 2022 01:30:38 GMT
Content-Type: text/html
Content-Length: 146
Connection: keep-alive
Server: nginx
Vary: Accept-Encoding

' ' ' '

However, adding "Mozilla" as a user agent will result in a 200.
' ' ' '
$ curl -ILs -A "Mozilla" https://www.geekforgeeks.com
HTTP/1.1 302 Found
Date: Sat, 29 Oct 2022 01:31:01 GMT
Server: Apache/2.4.38 (Debian)
Set-Cookie: __tad=1667007061.7580214; expires=Tue, 26-Oct-2032 01:31:01 GMT; Max-Age=315360000
Location: http://ww38.geekforgeeks.com/?subid1=20221029-1231-01df-8509-74ed46dc8a35
Connection: close
Content-Type: text/html; charset=UTF-8

HTTP/1.1 200 OK
Date: Sat, 29 Oct 2022 01:31:01 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Server: nginx
Vary: Accept-Encoding
Vary: Accept-Encoding
X-Adblock-Key: MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALquDFETXRn0Hr05fUP7EJT77xYnPmRbpMy4vk8KYiHnkNpednjOANJcaXDXcKQJN0nXKZJL7TciJD8AoHXK158CAwEAAQ==_MK0o9q2JhJUQNjBpBR2Ei5OAXCN8SHvGUFEOe8mpFQWgebMK/UCWCW+DrePVfic7NWSp6COzpQNDclVYFKhgdA==
Accept-CH: viewport-width
Accept-CH: dpr
Accept-CH: device-memory
Accept-CH: rtt
Accept-CH: downlink
Accept-CH: ect
Accept-CH: ua
Accept-CH: ua-full-version
Accept-CH: ua-platform
Accept-CH: ua-platform-version
Accept-CH: ua-arch
Accept-CH: ua-model
Accept-CH: ua-mobile
Accept-CH-Lifetime: 30

' ' ' '

[JMNewsome]

Carvana will reject a request without a user agent

curl -IL https://carvana.com
HTTP/1.1 403 Forbidden
Date: Sun, 06 Nov 2022 03:06:34 GMT
Content-Type: text/plain; charset=UTF-8
Content-Length: 16
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
Referrer-Policy: same-origin
Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Thu, 01 Jan 1970 00:00:01 GMT
Server: cloudflare
CF-RAY: 765a9d2caf3058c0-IAD

But will accept the request when using a user agent

curl -IL -A "Mozilla" https://carvana.com
HTTP/1.1 301 Moved Permanently
Date: Sun, 06 Nov 2022 03:06:45 GMT
Connection: keep-alive
Cache-Control: max-age=3600
Expires: Sun, 06 Nov 2022 04:06:45 GMT
Location: https://www.carvana.com/
Server: cloudflare
CF-RAY: 765a9d733df15ae1-IAD

HTTP/1.1 200 OK
Date: Sun, 06 Nov 2022 03:06:45 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Vary: Accept-Encoding
Vary: Accept-Encoding
Vary: Accept-Encoding
X-Powered-By: Express
Cache-Control: max-age=0
Pragma: no-cache
Expires: Sun, 06 Nov 2022 03:06:45 GMT
Set-Cookie: ai-chat-hand-raising=control; Expires=Sun Nov 06 2022 03:21:45 GMT+0000 (Coordinated Universal Time); path=/
Set-Cookie: aa-test-with-bcid=control; Expires=Sun Nov 06 2022 03:21:45 GMT+0000 (Coordinated Universal Time); path=/
Set-Cookie: mdt-home-next-gen=taxseason; Expires=Sun Nov 06 2022 03:21:45 GMT+0000 (Coordinated Universal Time); path=/
Set-Cookie: home-srch-vdp-affordability=bodystyle; Expires=Sun Nov 06 2022 03:21:45 GMT+0000 (Coordinated Universal Time); path=/
Set-Cookie: home-srch-jj-tiles-2022=experiment; Expires=Sun Nov 06 2022 03:21:45 GMT+0000 (Coordinated Universal Time); path=/
Set-Cookie: home-greatdeal-tax=experiment; Expires=Sun Nov 06 2022 03:21:45 GMT+0000 (Coordinated Universal Time); path=/
Set-Cookie: home-q4-2022-new-events-ramp-up=treatment; Expires=Sun Nov 06 2022 03:21:45 GMT+0000 (Coordinated Universal Time); path=/
Set-Cookie: BrowserCookieId=92ee22df-b8b0-cbb4-f5c1-2eed8bb9d0de; Path=/; Max-Age=31536000;
Set-Cookie: cdp_auid=92ee22df-b8b0-cbb4-f5c1-2eed8bb9d0de; Path=/; Domain=.carvana.com; Max-Age=63072000; Secure; HttpOnly; SameSite=None;
Strict-Transport-Security: max-age=31536000; includeSubDomains
CF-Cache-Status: DYNAMIC
Server: cloudflare
CF-RAY: 765a9d740ecc6fad-IAD