403 on some URLs with mdlc and curl, but not in browser?

[chris-dura]

I didn't want to open an issue, because we probably can't account for every wonky server setup on external sites. But, I'm just curious about how these requests work, and if anyone could guess at what's going on?

In our design system docs, we link out to a few pages on the popular Abstract and Figma docs, and few random sites, and they return as 403; however, I have no problem going to them in my browser.

• https://help.figma.com/hc/en-us/articles/360039823654-Download-the-Figma-desktop-app
• https://help.abstract.com/hc/en-us/categories/360005436612-Branches
• https://www.mkyong.com/android/android-radio-buttons-example

The curl commands all return HTTP/2 403 or HTTP/1.1 403 Forbidden.

curl -v https://help.figma.com/hc/en-us/articles/360039823654-Download-the-Figma-desktop-app -o /dev/null

curl -v https://help.abstract.com/hc/en-us/categories/360005436612-Branches -o /dev/null

curl -v https://www.mkyong.com/android/android-radio-buttons-example -o /dev/null

I know there was an issue with user-agent strings in the past... #159 (comment)

I wonder if this is just another case of sites not liking particular user agents?

[jeertmans]

Hello @chris-dura, I can confirm I have the same issue with this URL: https://faq.dailymotion.com/hc/en-us/articles/360022841393-How-to-preserve-the-player-aspect-ratio-on-a-responsive-page.

Any idea how to avoid the 403 status?

[chris-dura]

@jeertmans -- short answer: no.

It's most likely due to the target site's server setup. Per this comment, the target site probably refuses connections from "suspicious" user-agent strings, which makes total sense.

If that's truly the case, then the only "workaround" would be for "link checker" tools (like this one) to "spoof" their self-reported user-agent string to match a common one, like Mozilla... however, at this point, browsers and servers are also probably too smart for that as well. 🤷🏻

[jeertmans]

Thanks for the update!

Well I just disabled the URL check on that very specific case, hopefully it will never break ^^'