Removed firecrawl-py, fixed and improved firecrawl tool

[ahasasjeb]

Checklist:

Important

Please review the checklist below before submitting your pull request.

• Please open an issue before creating a PR or link to an existing issue
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

Description

Improved code, str will appear in the error Fixes #5897

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update, included: Dify Document
• Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement
• Dependency upgrade

Testing Instructions

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• Test A Run docker,use firecrawl,url is https://www.163.com/news/article/J64CN5IM000189FH.html?clickfrom=w_yw

• Test B

[ahasasjeb]

#5897

[ahasasjeb]

Fireclaw-py uses APGL license

[laipz8200]

I did a preliminary review of your code and found a few issues. Could you please address them first? Also, I would prefer if we could use type annotations in the method parameters.

[ahasasjeb]

I forgot to make the changes. After running it successfully, I submitted it directly and will make the changes in three hours ---- Replied Message ---- From ***@***.***> Date 07/03/2024 09:03 To ***@***.***> Cc ***@***.***>***@***.***> Subject Re: [langgenius/dify] Removed firecrawl-py, fixed and improved firecrawl tool (PR #5896) @laipz8200 requested changes on this pull request. I did a preliminary review of your code and found a few issues. Could you please address them first? Also, I would prefer if we could use type annotations in the method parameters. In api/core/tools/provider/builtin/firecrawl/firecrawl_appx.py:

class FirecrawlApp: - def __init__(self, api_key=None, base_url=None): + def __init__(self, api_key=None, api_url=None):

There's no need to change the variable names; the original ones are more suitable. In api/core/tools/provider/builtin/firecrawl/firecrawl_appx.py:

self.api_key = api_key - self.base_url = base_url or 'https://api.firecrawl.dev' - if self.api_key is None and self.base_url == 'https://api.firecrawl.dev': - raise ValueError('No API key provided') + if not self.api_key:

The self-hosted version doesn't require an API key. Making this change could cause errors. In api/core/tools/provider/builtin/firecrawl/tools/crawl.py:

crawl_result = app.crawl_url( url=tool_parameters['url'], - params=options,

Where are we handling options now? In api/core/tools/provider/builtin/firecrawl/firecrawl_appx.py:

- def _prepare_headers(self): - return { - 'Content-Type': 'application/json', - 'Authorization': f'Bearer {self.api_key}' - } - - def _post_request(self, url, data, headers): - return requests.post(url, headers=headers, json=data) + def crawl_url(self, url, wait=False, poll_interval=5, idempotency_key=None, **kwargs): + endpoint = f'{self.api_url}/v0/crawl' + headers = self._prepare_headers(idempotency_key) + data = {'url': url, **kwargs} + response = self._request('POST', endpoint, data, headers) + job_id = response['jobId'] # 确保使用正确的键名

Please avoid using non-English comments in the code. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[crazywoola]

Please resolve this comments by clicking those buttons.

[ahasasjeb]

ok

[laipz8200]

LGTM