Failing to retrieve BBC webpages with fetch

[dzx-dzx]

Hi all.

For the past week I have been investigating DIYgod/RSSHub#16032 (which is not created by me). This issue was about an user failing to fetch webpages from the BBC website with the error 403 Forbidden Edge 5. I found myself able to replicate the problem, but unable to fix it. Specifically, running

await fetch("https://www.bbc.com/news/articles/cw0ygnyxgnqo")

with node consistently returns a 403 status code and the Forbidden Edge 5 status text.

The following is attempts that I made.

Mimicing the browser

I can visit the BBC and any of its articles from my browser, so the headers were my first consideration. I copyed the fetch command from the dev tool and pasted it in node, but to no avail.

Trying on different devices

I tried:

• Node v20.11.1 & v22.2.0 running on Windows 11
• Node v22.2.0 on WSL Ubuntu
• Node v22.4.1 on native Linux Mint
• Node v22.4.1 on native Ubuntu

but all failed. Surprisingly, my phone works:

• Node v22.4.0 from Termux on native Android

Trying other tools

I tried python requests and curl, and both can work.

I also visited https://npm.runkit.com/ which works, though it only offers up to v18.

I know it's highly improbable that there's actually an issue with node and its native fetch itself. But I really have exhausted all my ideas. I'd be very grateful if anyone can help!

[mcollina]

They are doing some kind of user-agent sniffing, and they don't like ours. This will work:

await fetch("https://www.bbc.com/news/articles/cw0ygnyxgnqo", { headers: { 'User-Agent': 'curl/8.4.0' } })

[dzx-dzx]

I give it a try. On my Ubuntu it will still fail on the first attempt, but succeed on the second. I don't think fetch stored any thing between the two request...?
Other devices still don't work no matter how I repeated.

[mcollina]

we keep the connection alive if possible

[dzx-dzx]

I see...Thanks!