feat: better newUrlFunction for ProxyConfiguration

[barjin]

Based on changes from #2348 , this PR simplifies the proxy handling in the browser crawlers and makes those more intuitive.

Ready for reviews!

Closes #2065

[barjin]

Also cc #2310 - this solves the issue from there - but only for proxyUrls.

[vladfrangu]

Looks good to me otherwise 👀

[B4nan]

just a few nits/ideas

[barjin]

Looking at apify-sdk's ProxyConfiguration, the implementation there should be quite straightforward once we merge this - we inherit Crawlee's ProxyConfiguration class and can call the protected _handleTieredUrl() method with all the logic from there. Any ideas abt how we can make it even more seamless now here?

[harshmaur]

@barjin documentation for this ? How do we configure it?

[barjin]

@harshmaur - keep in mind that this feature is not yet in the stable version of Crawlee (will be included in the next release). Right now, you can try it out by installing the next tag (npm install crawlee@next).

These are mostly internal changes to how we manage proxies in browser-based crawlers, but the external API changed a bit too. See the JSDoc for newUrlFunction - you can now return null from the function if you want the crawler to not use proxy for this request. Also, the newUrlFunction is now called with (optional) request object. All in all, you can do something like this:

...
newUrlFunction: (sessionId, { request } = {}) {
   if (request?.url.includes('example.com')) return null;
   return 'default.proxy.com';
}
...

Let us know if you encounter any issues with this. Cheers!

[harshmaur]

Okay this is great! that's exactly what I needed.

…

Message ID: ***@***.***>