feat: Spider web scraper & crawler tool

[WilliamEspegren]

Description

Add Spider, the fastest open source scraper & crawler that returns LLM-ready data.

Twitter: @WilliamEspegren

Type of Change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)

How Has This Been Tested?

I have created a new workflow in the UI then tested it manually with all the different parameters.

Suggested Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

This PR was inspired by #3819

[WilliamEspegren]

@bowenliang123 could you check this one out? Let me know if you want any changes or help otherwise :)

[takatost]

Hi @WilliamEspegren, I tried out Spider, but neither the SIGN IN nor REGISTER buttons are bringing up the login screen as expected. It's happening on both Chrome and Safari.

[WilliamEspegren]

Hi @WilliamEspegren, I tried out Spider, but neither the SIGN IN nor REGISTER buttons are bringing up the login screen as expected. It's happening on both Chrome and Safari.

Hi @TakaTos! Thank you for mentioning this issue. Here is the link instead: https://spider.cloud/login

[WilliamEspegren]

@crazywoola @takatost just checking the status on this, if you want any help just ping me :)

[takatost]

Thanks!!

[ivuai]

@WilliamEspegren is this tool still working in latest version? because it doesn't work for me when I enter API key it gives me this error: argument of type 'NoneType' is not iterable

[WilliamEspegren]

Hey @ivuai! How does your parameters look like, and what nodes are you using?

[ivuai]

I don't use any node, I'm just simply trying to add API key in tools and I don't use any parameters. Here is the screenshot.

[WilliamEspegren]

Thank you @ivuai! Weird, I will look and test it out tomorrow :)

[WilliamEspegren]

@ivuai the API was somehow down. I have added error checks to display this more clearly if that ever happens again, thanks for mentioning this 👏