Redirects 503 and ERR

[reagle]

I've long used linkcheckers but only recently encountered lychee and it is fantastic. It's informative, fast, and can run on the original markdown itself! Thank you!

I do have a question about the following issues encountered in checking a syllabus.

1. Does the DOI checker block lychee?
2. I think I'm using the canonical URLs for the datasociety.net pages. What's going on?

[cda-syllabus-FA.md]:
✗ [ERR] https://points.datasociety.net/agnotology-and-epistemological-fragmentation-56aa3c509c6b | Failed: Network error: error following redirect for url (https://points.datasociety.net/agnotology-and-epistemological-fragmentation-56aa3c509c6b?gi=f14b2c450f03): too many redirects
✗ [503] http://dx.doi.org/10.1002/hbe2.196 | Failed: Network error: Service Unavailable
✗ [503] http://dx.doi.org/10.1177/14614448221099234 | Failed: Network error: Service Unavailable
✗ [ERR] https://theblog.okcupid.com/the-big-lies-people-tell-in-online-dating-a9e3990d6ae2 | Failed: Network error: error following redirect for url (https://theblog.okcupid.com/the-big-lies-people-tell-in-online-dating-a9e3990d6ae2?gi=e8725fc33855): too many redirects
✗ [ERR] https://www.proquest.com/docview/2412452381?accountid=12826&pq-origsite=primo | Failed: Network error: error following redirect for url (https://www.proquest.com/docview/2412452381?accountid=12826&forcedol=true&forcedol=true&forcedol=true&pq-origsite=primo&forcedol=true): too many redirects
✗ [ERR] https://points.datasociety.net/did-media-literacy-backfire-7418c084d88d#.d46kox6e1 | Failed: Network error: error following redirect for url (https://points.datasociety.net/did-media-literacy-backfire-7418c084d88d?gi=da45da0550dc): too many redirects
✗ [503] http://dx.doi.org/10.1177/15327086211029357 | Failed: Network error: Service Unavailable

🔍 109 Total ✅ 102 OK 🚫 7 Errors (HTTP:7)

[mre]

I've long used linkcheckers but only recently encountered lychee and it is fantastic. It's informative, fast, and can run on the original markdown itself! Thank you!

Thanks! Such feedback always makes us happy. ☺️

Something's going on there. curl has the same issues:

> curl -L https://points.datasociety.net/agnotology-and-epistemological-fragmentat
ion-56aa3c509c6b
curl: (47) Maximum (50) redirects followed

[mre]

No clue what's up with the 503, though.

echo 'http://dx.doi.org/10.1002/hbe2.196' | lychee -XHEAD --user-agent 'curl/7.7
9.1' -- -
✗ [503] http://dx.doi.org/10.1002/hbe2.196 | Failed: Network error: Service Unavailabl

[mre]

It works in curl. 🤔

[reagle]

On the DOI, wget -S does a redirect from http://dx.doi.org/10.1002/hbe2.196 to https://onlinelibrary.wiley.com/doi/10.1002/hbe2.196 but then gets a 503. I haven't tried different user agent strings to see if that's the problem.

[mre]

It definitely has something to do with the way the server handles the request.
I built https://github.com/lycheeverse/geturl-test to test if impersonating a Chrome browser works (it's based on https://github.com/4JX/reqwest-impersonate) and it does.

> cargo run --no-default-features --features curl -- 'http://dx.doi.org/10.1002/hbe2
get_url: http://dx.doi.org/10.1002/hbe2.196
response: Response { url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: Non
e, host: Some(Domain("onlinelibrary.wiley.com")), port: None, path: "/action/cookieAbsent", query: No
ne, fragment: None }, status: 200, headers: {"date": "Wed, 09 Nov 2022 14:15:39 GMT", "content-type":
 "text/html; charset=UTF-8", "cf-ray": "767729650a157162-DUS", "cache-control": "no-cache", "link": "
</wro/lpko~product.css>; rel=preload; as=style; nopush, </products/pericles/releasedAssets/css/build.
css>; rel=preload; as=style; nopush, <https://assets.adobedtm.com/>; rel=preconnect;, <https://wiley.
grapeshot.co.uk>; rel=preconnect;, <https://www.google-analytics.com>; rel=preconnect;, <https://www.
googletagmanager.com>; rel=preconnect;, <https://www.googleadservices.com>; rel=preconnect;", "set-co
okie": "JSESSIONID=aaaXHufQdhUpqbm1h_yry; path=/; secure; SameSite=None; HttpOnly", "set-cookie": "__
cf_bm=i7WarTfqL38NbyV.3ovWMXUe12FbBbDCeJsgUHngqoA-1668003339-0-AVVJlYs3B5pW/YUtm7FTqPOfUj6nQyscNA3hGz
DJGTX4UGwm7h0WgKToBmiXZrNWO78iXfApL8NG+MbsTftk19NH/ZG/guJ3JWSJAq0gTq60; path=/; expires=Wed, 09-Nov-2
2 14:45:39 GMT; domain=.onlinelibrary.wiley.com; HttpOnly; Secure; SameSite=None", "strict-transport-
security": "max-age=15552000", "cf-cache-status": "DYNAMIC", "content-security-policy": "frame-ancest
ors https://onlinelibrary.wiley.com", "x-content-type-options": "nosniff", "x-frame-options": "ALLOW-
FROM https://onlinelibrary.wiley.com", "x-webstats-respid": "4236ffe2770760f9d20b13fdb6cf4e7e", "x-xs
s-protection": "1; mode=block", "vary": "Accept-Encoding", "server": "cloudflare"} }
status: 200 OK

We saw something similar in #733.

[mre]

Oh, the above command I used was actually

cargo run --no-default-features --features reqwest_impersonate -- 'http://dx.doi.o
rg/10.1002/hbe2.196'

--features curl also produced a 503

[reagle]

BTW: I quickly checked the DOI spec, it says nothing specific about this, and the FAQ, which mentions they use CloudFlare. Apparently you can email them at contact@doi.org.

[mre]

I'm still on the fence on whether we should integrate reqwest-impersonate as an alternative backend. It seems to resolve some of the gnarlier edge-cases, even though I'm worried it might become a maintenance burden in the long run if reqwest and reqwest-impersonate diverge. Open for opinions.

[reagle]

I understand. There's always a few poorly configured servers in any link checking. The DOI one bugs me, though, because their reason for existing is to resolve stable URLs. It'd be great if we could help them do the right thing rather than lychee having to do something complicated to accommodate them.

[mre]

I sent out an email to the email address you provided and added you to CC.

[reagle]

Just finished updating my spring syllabi and checked my links and see this is still an issue. The DOI folks never got back to you?

[cda-syllabus-SP.md]:
✗ [503] http://dx.doi.org/10.1002/hbe2.196 | Failed: Network error: Service Unavailable
✗ [403] https://chat.openai.com/chat | Failed: Network error: Forbidden
✗ [503] http://dx.doi.org/10.1177/15327086211029357 | Failed: Network error: Service Unavailable

[mre]

The DOI folks did reply, however they redirected me to other maintainers. Need to follow up with them.

[mre]

Follow-up email sent.

[mre]

I got a reply from crossref.org in which I got redirected to Wiley.com, which is responsible for this particular error message. However it seems I can only contact them by submitting a case here and I would need to create an account with them. It's not entirely clear to me where the message would end up at and I honestly thing the entire process is a bit tedious.

If someone wants to follow through, here is the message template I was planning to use with all the information I have so far:

I noticed that DOI links cannot be checked at the moment with
our tool.
Specifically, links to http://onlinelibrary.wiley.com/ fail with both
lychee and curl:

curl -i 'https://onlinelibrary.wiley.com/doi/10.1002/hbe2.196'
HTTP/2 503

The same happens when following redirects from the original URL of course,

curl -iL http://dx.doi.org/10.1002/hbe2.196
...
HTTP/2 503

The above request redirects until the final 503.

I think it has something to do with Cloudflare's bot detection,
because the final response body contains the following:

<h2 class="h2" id="challenge-running">
Checking if the site connection is secure
</h2>
<noscript>
<div id="challenge-error-title">
<div class="h2">
<span class="icon-wrapper">
<div class="heading-icon warning-icon"></div>
</span>
<span id="challenge-error-text">
Enable JavaScript and cookies to continue
</span>
</div>
</div>
</noscript>

Seems like they do some browser detection, which is something that
neither curl or lychee can pass.
I'm guessing they have "Bot Fight Mode" on in Cloudflare:
https://developers.cloudflare.com/bots/get-started/free/

I was wondering if you have someone from wiley.com that I could talk to
about this issue, because it potentially affects a lot of users
and might take a simple configuration change to fix.

[lebensterben]

FYI, it works fine here. (on master branch)

> cargo run -- http://dx.doi.org/10.1002/hbe2.196
    Finished dev [unoptimized + debuginfo] target(s) in 0.16s
     Running `target/debug/lychee 'http://dx.doi.org/10.1002/hbe2.196'`
🔍 1 Total ✅ 1 OK 🚫 0 Errors

[mre]

Strangely, this doesn't work, which is what OP is running into:

echo 'http://dx.doi.org/10.1002/hbe2.196' | lychee -

[reagle]

Likely to be a futile gesture of perverse perseverance but I submitted case 17999077.

[mre]

Haha, nice one. The formatting is a bit off, but you got the message across. 😅