diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 491db657a..132af1255 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -93,6 +93,56 @@ This part outlines the guidelines and best practices for conducting code reviews Code reviews are an essential part of maintaining the quality and integrity of our open source project. By following these guidelines, we can ensure that CAMEL remains robust, secure, and easy to maintain, while also fostering a collaborative and welcoming community. +### Guideline for Writing Docstrings + +This guideline will help you write clear, concise, and structured docstrings for contributing to `CAMEL`. + +#### 1. Use the Triple-Quoted String with `r"""` (Raw String) +Begin the docstring with `r"""` to indicate a raw docstring. This prevents any issues with special characters and ensures consistent formatting, especially in documentation tools like Sphinx. + +#### 2. Provide a Brief Class or Method Description +- Start with a concise summary of the purpose and functionality. +- Keep each line under `79` characters. +- The summary should start on the first line without a linebreak. + +Example: +```python +r"""Class for managing conversations of CAMEL Chat Agents. +""" +``` + +#### 3. Document Parameters in the Args Section +- Use an `Args`: section for documenting constructor or function parameters. +- Maintain the `79`-character limit for each line, and indent continuation lines by 4 spaces. +- Follow this structure: + - Parameter Name: Match the function signature. + - Type: Include the type (e.g., `int`, `str`, custom types like `BaseModelBackend`). + - Description: Provide a brief explanation of the parameter's role. + - Default Value: Use (`default: :obj:`) to indicate default values. + +Example: +```markdown +Args: + system_message (BaseMessage): The system message for initializing + the agent's conversation context. + model (BaseModelBackend, optional): The model backend to use for + response generation. Defaults to :obj:`OpenAIModel` with + `GPT_4O_MINI`. (default: :obj:`OpenAIModel` with `GPT_4O_MINI`) +``` + +### Naming Principle šŸ›”ļø +#### Avoid Abbreviations in Naming +- Abbreviations can lead to ambiguity, especially since variable names and code in CAMEL are directly used by agents. +- Use clear, descriptive names that convey meaning without requiring additional explanation. This improves both human readability and the agent's ability to interpret the code. + +Examples: + +- Bad: msg_win_sz +- Good: message_window_size + +By adhering to this principle, we ensure that CAMEL remains accessible and unambiguous for both developers and AI agents. + + ### Board Item Create Workflow šŸ› ļø At CAMEL, we manage our project through a structured workflow that ensures efficiency and clarity in our development process. Our workflow includes stages for issue creation and pull requests (PRs), sprint planning, and reviews. diff --git a/README.md b/README.md index 1fa635a6a..25164663f 100644 --- a/README.md +++ b/README.md @@ -164,9 +164,10 @@ Detailed guidance can be find [here](https://github.com/camel-ai/camel/blob/mast By default, the agent uses the `ModelType.DEFAULT` model from the `ModelPlatformType.DEFAULT`. You can configure the default model platform and model type using environment variables. If these are not set, the agent will fall back to the default settings: -- `ModelPlatformType.DEFAULT = "openai"` - -- `ModelType.DEFAULT = "gpt-4o-mini"` +```bash +ModelPlatformType.DEFAULT = "openai" +ModelType.DEFAULT = "gpt-4o-mini" +``` ### Setting Default Model Platform and Model Type (Optional) @@ -346,6 +347,8 @@ For more details, please see our [`Models Documentation`](https://docs.camel-ai. We implemented amazing research ideas from other works for you to build, compare and customize your agents. If you use any of these modules, please kindly cite the original works: - `TaskCreationAgent`, `TaskPrioritizationAgent` and `BabyAGI` from *Nakajima et al.*: [Task-Driven Autonomous Agent](https://yoheinakajima.com/task-driven-autonomous-agent-utilizing-gpt-4-pinecone-and-langchain-for-diverse-applications/). [[Example](https://github.com/camel-ai/camel/blob/master/examples/ai_society/babyagi_playing.py)] +- `PersonaHub` from *Tao Ge et al.*: [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://arxiv.org/pdf/2406.20094). [[Example](https://github.com/camel-ai/camel/blob/master/examples/personas/personas_generation.py)] + ## Other Research Works Based on Camel - [Agent Trust](http://agent-trust.camel-ai.org/): Can Large Language Model Agents Simulate Human Trust Behavior? diff --git a/camel/personas/persona.py b/camel/personas/persona.py index b1201d85d..37a7b2a22 100644 --- a/camel/personas/persona.py +++ b/camel/personas/persona.py @@ -11,6 +11,7 @@ # See the License for the specific language governing permissions and # limitations under the License. # =========== Copyright 2023 @ CAMEL-AI.org. All Rights Reserved. =========== +import json import uuid from typing import ClassVar, Optional, Union @@ -25,10 +26,17 @@ class Persona(BaseModel): Attributes: name (Optional[str]): Name of the persona. description (Optional[str]): Description of the persona. - t2p_prompt (Union[TextPrompt, str]): Text to Persona Prompt. - p2p_prompt (Union[TextPrompt, str]): Persona to Persona Prompt. + text_to_persona_prompt (Union[TextPrompt, str]): The prompt to convert + text into a persona. + persona_to_persona_prompt (Union[TextPrompt, str]): Persona-to-Persona + interaction prompt. id (uuid.UUID): The unique identifier for the persona, automatically generated. + _id (uuid.UUID): Internal unique identifier for the persona, + generated lazily using `uuid.uuid4`. + model_config (ClassVar[ConfigDict]): Configuration for the Pydantic + model. Allows arbitrary types and includes custom JSON schema + settings. """ name: Optional[str] = None @@ -37,13 +45,14 @@ class Persona(BaseModel): # Field with default_factory to avoid circular import issues # Union type allows either TextPrompt or str - t2p_prompt: Union[TextPrompt, str] = Field( + text_to_persona_prompt: Union[TextPrompt, str] = Field( default_factory=lambda: PersonaHubPrompt.TEXT_TO_PERSONA, description="Text to Persona Prompt", ) - # Similar to t2p_prompt, using default_factory for lazy evaluation - p2p_prompt: Union[TextPrompt, str] = Field( + # Similar to text_to_persona_prompt, using default_factory for lazy + # evaluation + persona_to_persona_prompt: Union[TextPrompt, str] = Field( default_factory=lambda: PersonaHubPrompt.PERSONA_TO_PERSONA, description="Persona to Persona Prompt", ) @@ -56,10 +65,10 @@ class Persona(BaseModel): # Custom JSON schema configuration json_schema_extra={ "properties": { - # Ensure t2p_prompt and p2p_prompt are treated as strings in - # JSON schema - "t2p_prompt": {"type": "string"}, - "p2p_prompt": {"type": "string"}, + # Ensure text_to_persona_prompt and persona_to_persona_prompt + # are treated as strings in JSON schema + "text_to_persona_prompt": {"type": "string"}, + "persona_to_persona_prompt": {"type": "string"}, } }, ) @@ -75,12 +84,20 @@ def model_json_schema(cls): return schema def dict(self, *args, **kwargs): - # Output: {'name': 'Alice', 'description': None, 't2p_prompt': '...', 'p2p_prompt': '...', 'id': 'f47ac10b-58cc-4372-a567-0e02b2c3d479'} # noqa: E501 + # Output: {'name': 'Alice', 'description': None, 'text_to_persona_prompt': '...', 'persona_to_persona_prompt': '...', 'id': 'f47ac10b-58cc-4372-a567-0e02b2c3d479'} # noqa: E501 d = super().model_dump(*args, **kwargs) d['id'] = str(self.id) return d def json(self, *args, **kwargs): - # Output: '{"name": "Alice", "description": null, "t2p_prompt": "...", "p2p_prompt": "...", "id": "f47ac10b-58cc-4372-a567-0e02b2c3d479"}' # noqa: E501 + # Output: '{"name": "Alice", "description": null, "text_to_persona_prompt": "...", "persona_to_persona_prompt": "...", "id": "f47ac10b-58cc-4372-a567-0e02b2c3d479"}' # noqa: E501 d = self.dict(*args, **kwargs) - return super().json(d, *args, **kwargs) + return json.dumps( + d, + indent=4, # Pretty-print with 4 spaces indentation + sort_keys=True, # Sort keys alphabetically + separators=( + ",", + ": ", + ), # Fine-tune separators for better readability + ) diff --git a/camel/personas/persona_hub.py b/camel/personas/persona_hub.py index 11c285f5f..595331e79 100644 --- a/camel/personas/persona_hub.py +++ b/camel/personas/persona_hub.py @@ -31,12 +31,16 @@ class PersonaResponse(BaseModel): persona_name: str = Field(description="The name of the persona") persona_description: str = Field( - description="The description of the persona" + description="The description of the persona." ) class PersonaHub: - r"""PersonaHub proposes a novel persona-driven data synthesis methodology + r"""The PersonaHub adapted from `"Scaling Synthetic Data Creation with 1, + 000,000,000 Personas" + `_. + + PersonaHub proposes a novel persona-driven data synthesis methodology that leverages various perspectives within a large language model (LLM) to create diverse synthetic data. By showcasing PersonaHub's use cases in synthesizing high-quality mathematical and logical reasoning problems, @@ -45,7 +49,7 @@ class PersonaHub: synthesis is versatile, scalable, flexible, and easy to use, potentially driving a paradigm shift in synthetic data creation and applications in practice, which may have a profound impact on LLM research and development. - Please refer to the paper for more details: https://arxiv.org/pdf/2406.20094 + Please refer to the paper for more details: https://arxiv.org/pdf/2406.20094. Args: model (BaseModelBackend, optional): The model to use for persona @@ -76,7 +80,7 @@ def __delitem__(self, persona_id: uuid.UUID): if persona_id in self.personas: del self.personas[persona_id] else: - raise KeyError("Persona ID not found") + raise KeyError("Persona ID not found.") def __getitem__(self, persona_id: uuid.UUID) -> Persona: r"""Get a persona by ID. @@ -87,7 +91,7 @@ def __getitem__(self, persona_id: uuid.UUID) -> Persona: if persona_id in self.personas: return self.personas[persona_id] else: - raise KeyError("Persona ID not found") + raise KeyError("Persona ID not found.") def text_to_persona( self, @@ -107,8 +111,12 @@ def text_to_persona( """ persona = Persona() - t2p_prompt: Union[TextPrompt, str] = persona.t2p_prompt - t2p_prompt_instruction = t2p_prompt.format(action=action, text=text) + text_to_persona_prompt: Union[TextPrompt, str] = ( + persona.text_to_persona_prompt + ) + text_to_persona_prompt_instruction = text_to_persona_prompt.format( + action=action, text=text + ) # Set Agent to generate personal t2p_agent = ChatAgent( @@ -119,7 +127,7 @@ def text_to_persona( # Get output from agent try: response = t2p_agent.step( - t2p_prompt_instruction, + text_to_persona_prompt_instruction, response_format=PersonaResponse, # type: ignore[arg-type] ) parsed_content = ast.literal_eval(response.msg.content) @@ -141,7 +149,9 @@ def persona_to_persona(self, persona: Persona) -> Dict[uuid.UUID, Persona]: Returns: Dict[uuid.UUID, Persona]: A dictionary of related personas. """ - p2p_prompt: Union[TextPrompt, str] = persona.p2p_prompt + persona_to_persona_prompt: Union[TextPrompt, str] = ( + persona.persona_to_persona_prompt + ) answer_template = """ You MUST answer the question according to the format of the ANSWER TEMPLATE, and you can only modify the content within . ===== ANSWER TEMPLATE ===== @@ -151,8 +161,8 @@ def persona_to_persona(self, persona: Persona) -> Dict[uuid.UUID, Persona]: n. persona_name: persona_description: """ # noqa: E501 - p2p_prompt_instruction = ( - p2p_prompt.format( + persona_to_persona_prompt_instruction = ( + persona_to_persona_prompt.format( persona_name=persona.name, persona_description=persona.description, ) @@ -167,7 +177,7 @@ def persona_to_persona(self, persona: Persona) -> Dict[uuid.UUID, Persona]: # Get output from agent try: response = p2p_agent.step( - p2p_prompt_instruction # type: ignore[arg-type] + persona_to_persona_prompt_instruction # type: ignore[arg-type] ) # Structured output (TODO: Use a more robust parser) pattern = r"(\d+)\.\s*persona_name:\s*(.*?)\s*persona_description:\s*(.*?)\s*(?=\d+\.|$)" # noqa: E501 diff --git a/docs/cookbooks/ingest_data_from_websites_with_Firecrawl.ipynb b/docs/cookbooks/ingest_data_from_websites_with_Firecrawl.ipynb new file mode 100644 index 000000000..1bb6a4942 --- /dev/null +++ b/docs/cookbooks/ingest_data_from_websites_with_Firecrawl.ipynb @@ -0,0 +1,754 @@ +{ + "cells": [ + { + "attachments": {}, + "cell_type": "markdown", + "metadata": { + "id": "1Nj0Oqnoy6oJ" + }, + "source": [ + "\n", + "# 3 ways to ingest data from websites with Firecrawl" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "You can also check this cookbook in colab [here](https://colab.research.google.com/drive/1lOmM3VmgR1hLwDKdeLGFve_75RFW0R9I?usp=sharing)\n", + "\n", + "In this blog, weā€™ll introduce Firecrawl, a versatile web scraping and crawling tool designed to extract data efficiently from websites, which has been integrated with CAMEL. Today weā€™ll walk through three key featuresā€”Scrape, Crawl, and Mapā€”each tailored with a CAMEL agent use case.\n", + "\n", + "## Table of Content:\n", + "\n", + "1. Introduction\n", + "2. šŸ”„ Firecrawl: To crawl\n", + "3. šŸ”„ Firecrawl: To Scrape\n", + "4. šŸ”„ Firecrawl: To Map\n", + "5. Conclusion\n", + "\n", + "\n", + "ā­ **Star the Repo**\n", + "\n", + "If you find CAMEL useful or interesting, please consider giving it a star on our [CAMEL GitHub Repo](https://github.com/camel-ai/camel)! Your stars help others find this project and motivate us to continue improving it.\n", + "\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "rrkB7ZbPDkrC" + }, + "source": [ + "\n", + "## **Introduction**\n", + "\n", + "**Firecrawl** developed by the Mendable.ai team, is a data ingestion tool that streamlines web data extraction using web scraping, API access, and automated browser interactions. Itā€™s ideal for collecting structured and unstructured data from websites for analytics.\n", + "\n", + "It effectively manages complex tasks such as handling reverse proxies, implementing caching strategies, adhering to rate limits, and accessing content blocked by JavaScript.\n", + "\n", + "### **Features of Firecrawl**:\n", + "\n", + "**Crawl**: Collects content from all URLs within a web page, converting it into an LLM-ready format for seamless analysis.\n", + "\n", + "**Scrape**: Extracts content from a single URL, delivering it in formats ready for LLMs, including markdown, structured data (via LLM Extract), screenshots, and HTML.\n", + "\n", + "**Map**: Inputs a website and retrieves all URLs associated with it at high speed, enabling a comprehensive and efficient site overview.\n", + "\n", + "\n", + "\n", + "All the above features make it ideal for collecting structured and unstructured data from websites for agentic workflows.\n", + "***CAMEL-AI has integrated Firecrawl to enhance its web data extraction capabilities***.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7p-JjpyNVcCT" + }, + "source": [ + "First, install the CAMEL package with all its dependencies and input the OPENAI API Key.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true, + "id": "I2X5A0LBc92C" + }, + "outputs": [], + "source": [ + "pip install \"camel-ai[all]==0.2.9\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "Jo-X1VQwUdIU", + "outputId": "c81423a1-9901-4652-e81a-fd26e289b446" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Enter your API key: Ā·Ā·Ā·Ā·Ā·Ā·Ā·Ā·Ā·Ā·\n" + ] + } + ], + "source": [ + "import os\n", + "from getpass import getpass\n", + "\n", + "# Prompt for the API key securely\n", + "openai_api_key = getpass('Enter your API key: ')\n", + "os.environ[\"OPENAI_API_KEY\"] = openai_api_key" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "0J0_iW-YVcq2" + }, + "source": [ + "## šŸ”„ **Firecrawl: To crawl**" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "LJLuJeLSehbr" + }, + "source": [ + "Let's get started with the exploration of the first feature of Firecrawl - Crawl: Extracts content from all subpages in an LLM-ready format (markdown, structured data, screenshot, HTML, links, metadata) for easy analysis.\n", + "\n", + "Step 1: Set up your firecrawl API key\n", + "\n", + "You just need to go to this link and sign in to get your API Key: https://www.firecrawl.dev/app/api-keys" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "DFW8uMF5dI9c", + "outputId": "ebdfdc49-e45e-439e-dd03-69e1bf3e5571" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Enter your API key: Ā·Ā·Ā·Ā·Ā·Ā·Ā·Ā·Ā·Ā·\n" + ] + } + ], + "source": [ + "import os\n", + "from getpass import getpass\n", + "\n", + "# Prompt for the Firecrawl API key securely\n", + "firecrawl_api_key = getpass('Enter your API key: ')\n", + "os.environ[\"FIRECRAWL_API_KEY\"] = firecrawl_api_key" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BuPVfcSDawrH" + }, + "source": [ + "Step 2: Import necessary modules" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "u9BC0ka0aZGH" + }, + "outputs": [], + "source": [ + "from camel.loaders import Firecrawl" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "F-GtoKDsShDl" + }, + "source": [ + "Step 3: Crawl the website\n", + "\n", + "It will crawl the CAMEL-AI website and generate the LLM-friendly output as shown in markdown below." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "_NcsNM9rc_LT", + "outputId": "e3e4de61-0afe-4fd1-a429-94db01c38db7" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "completed\n", + "Camel-AI Team\n", + "\n", + "# We are finding the scaling law of agent\n", + "\n", + "šŸ« CAMEL is an open-source library designed for the study of autonomous and communicative agents. We believe that studying these agents on a large scale offers valuable insights into their behaviors, capabilities, and potential risks. To facilitate research in this field, we implement and support various types of agents, tasks, prompts, models, and simulated environments.\n", + "\n", + "**We are** always looking for more **contributors** and **collaborators**. Contact us to join forces怂\n", + "\n", + "[Slack](https://join.slack.com/t/camel-kwr1314/shared_invite/zt-1vy8u9lbo-ZQmhIAyWSEfSwLCl2r2eKA) [Discord](https://discord.gg/CNcNpquyDc)\n", + "\n", + "#### All contributors\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830ee9ab51435525e8c57a_1.jpeg)[lightaime](https://github.com/lightaime)\n", + "\n", + "Commits:\n", + "\n", + "116\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830ef16f49819f7dfa140e_3.png)[Wendong-Fan](https://github.com/Wendong-Fan)\n", + "\n", + "Commits:\n", + "\n", + "75\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830ef7600c40531b5c74e2_4.png)[Obs01ete](https://github.com/Obs01ete)\n", + "\n", + "Commits:\n", + "\n", + "38\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830f2b0d41ffb78faa6c58_6.png)[dandansamax](https://github.com/dandansamax)\n", + "\n", + "Commits:\n", + "\n", + "35\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/6683d081f7586e59387edbb6_5.jpeg)[zechengz](https://github.com/zechengz)\n", + "\n", + "Commits:\n", + "\n", + "26\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830f30d401b408fa35facf_7.png)[hammoudhasan](https://github.com/hammoudhasan)\n", + "\n", + "Commits:\n", + "\n", + "18\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830f36c12354b7846c3a51_8.jpeg)[yiyiyi0817](https://github.com/yiyiyi0817)\n", + "\n", + "Commits:\n", + "\n", + "9\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830f3a7bce11b6dbe3e222_9.jpeg)[HalberdOfPineapple](https://github.com/HalberdOfPineapple)\n", + "\n", + "Commits:\n", + "\n", + "9\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830f3fb0c6b06595b037e4_10.jpeg)[ocss884](https://github.com/ocss884)\n", + "\n", + "Commits:\n", + "\n", + "9\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830f43f41f537d536c5f5f_11.png)[Benjamin-eecs](https://github.com/Benjamin-eecs)\n", + "\n", + "Commits:\n", + "\n", + "8\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830f479dfcbf180e4fd018_12.jpeg)[Appointat](https://github.com/Appointat)\n", + "\n", + "Commits:\n", + "\n", + "7\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830f4e9f8f288796edd2b2_13.png)[zhiyu-01](https://github.com/zhiyu-01)\n", + "\n", + "Commits:\n", + "\n", + "5\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830f58a5e4a625d5793931_14.jpeg)[rsrbk](https://github.com/rsrbk)\n", + "\n", + "Commits:\n", + "\n", + "4\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830f5d7bce11b6dbe3f728_15.png)[HaniItani](https://github.com/HaniItani)\n", + "\n", + "Commits:\n", + "\n", + "3\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830f63bf86deec6b7f1a09_16.png)[jjyaoao](https://github.com/jjyaoao)\n", + "\n", + "Commits:\n", + "\n", + "3\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830f69b1a115cbcfe49dc5_17.jpeg)[willshang76](https://github.com/willshang76)\n", + "\n", + "Commits:\n", + "\n", + "3\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830f6f1ee3fc17a837f847_18.jpeg)[hychen-naza](https://github.com/hychen-naza)\n", + "\n", + "Commits:\n", + "\n", + "2\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830f760db8fb00b4739fd2_19.png)[FUYICC](https://github.com/FUYICC)\n", + "\n", + "Commits:\n", + "\n", + "2\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830f7b524b86a3ebe633c1_20.jpeg)[koch3092](https://github.com/koch3092)\n", + "\n", + "Commits:\n", + "\n", + "2\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830f81853d6f6c8f09fe8d_21.png)[cat-searcher-bot](https://github.com/cat-searcher-bot)\n", + "\n", + "Commits:\n", + "\n", + "1\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830f86ef774fafa799e956_22.jpeg)[Undertone0809](https://github.com/Undertone0809)\n", + "\n", + "Commits:\n", + "\n", + "1\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830f8abe25cfeebf796c94_23.jpeg)[ArnoldIOI](https://github.com/ArnoldIOI)\n", + "\n", + "Commits:\n", + "\n", + "1\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830f9c699384adbc10e9f8_24.jpeg)[ZIYU-DEEP](https://github.com/ZIYU-DEEP)\n", + "\n", + "Commits:\n", + "\n", + "1\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830fa9ce07b2ac7dffeaef_25.jpeg)[duducheng](https://github.com/duducheng)\n", + "\n", + "Commits:\n", + "\n", + "1\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830fb440bef9732785a628_26.png)[forever-ly](https://github.com/forever-ly)\n", + "\n", + "Commits:\n", + "\n", + "1\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830ff2f41f537d536d020e_27.jpeg)[EddieFahrenheit](https://github.com/EddieFahrenheit)\n", + "\n", + "Commits:\n", + "\n", + "1\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830ff823bea7f626c0e3a7_28.png)[riverfor](https://github.com/riverfor)\n", + "\n", + "Commits:\n", + "\n", + "1\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66830ffd23bea7f626c0e7fe_29.jpeg)[raywhoelse](https://github.com/raywhoelse)\n", + "\n", + "Commits:\n", + "\n", + "1\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/6683100268f1349b3eb6c055_30.jpeg)[ZackYule](https://github.com/ZackYule)\n", + "\n", + "Commits:\n", + "\n", + "1\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/668310074683c2253ba26897_31.jpeg)[fengju0213](https://github.com/fengju0213)\n", + "\n", + "Commits:\n", + "\n", + "1\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/6683100b6b83da024d9eceff_32.jpeg)[Skevinci](https://github.com/Skevinci)\n", + "\n", + "Commits:\n", + "\n", + "1\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/6683100ea0d0bbb5aa97c1f1_33.jpeg)[eltociear](https://github.com/eltociear)\n", + "\n", + "Commits:\n", + "\n", + "1\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/66831013feedb63ddc545bf3_34.jpeg)[tczhangzhi](https://github.com/tczhangzhi)\n", + "\n", + "Commits:\n", + "\n", + "1\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/668310184a5c41506fd7f350_35.jpeg)[fbaldassarri](https://github.com/fbaldassarri)\n", + "\n", + "Commits:\n", + "\n", + "1\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/6683101c2dcf19fe9a1f95da_36.png)[zeromquan](https://github.com/zeromquan)\n", + "\n", + "Commits:\n", + "\n", + "1\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/6683101f524b86a3ebe6b127_37.png)[onemquan](https://github.com/onemquan)\n", + "\n", + "Commits:\n", + "\n", + "1\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a155491a54a40551bd7f/668310236c11244074a2aae8_38.jpeg)[zestor](https://github.com/zestor)\n", + "\n", + "Commits:\n", + "\n", + "1\n", + "\n", + "## Our Advisors\n", + "\n", + "[![](https://cdn.prod.website-files.com/6659a154491a54a40551bc78/66c6091430068e70376b2434_KAUST-CEMSE-EE-VCC-IVUL-Bernard-Ghanem-2045-edited-4k.jpg)\\\\\n", + "Bernard Ghanem\\\\\n", + "\\\\\n", + "King Abdullah University of Science and Technology\\\\\n", + "\\\\\n", + "- Rising star professor in the Middle East region \\\\\n", + "- He was Awarded the Shoman Arab Researcher Award for the topic of Machine Learning and Big Data\\\\\n", + "- Deputy Director of AI Initiative at Kaust and now leading a bigger initiative at Kaust on GenAI](https://cemse.kaust.edu.sa/people/person/bernard-ghanem) [![](https://cdn.prod.website-files.com/6659a154491a54a40551bc78/66c609ea3025b97ff20628a6_Philip_torr.jpg)\\\\\n", + "Philip Torr\\\\\n", + "\\\\\n", + "University of Oxford\\\\\n", + "\\\\\n", + "- 2019: Fellow of the Royal Academy of Engineering (FREng) in 2019\\\\\n", + "- 2021: Fellow of the Royal Society (FRS) for contributions to computer vision\\\\\n", + "- 2021: Turing AI world leading researcher fellow](https://eng.ox.ac.uk/people/philip-torr/)\n", + "\n", + "Get started\n", + "\n", + "## Build your multi-agent system today\n", + "\n", + "Get started by joining our community\n", + "\n", + "[Join Discord](http://discord.camel-ai.org/)\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a154491a54a40551bc78/6686afd48a4c823499baa2e7_Camel%20Logo%20Long.png)\n", + "\n", + "Finding the scaling law of agents\n", + "\n", + "![](https://cdn.prod.website-files.com/6659a154491a54a40551bc78/6659a155491a54a40551bd92_mail.svg)\n", + "\n", + "RESEARCH\n", + "\n", + "[CAMEL](https://github.com/camel-ai/camel) [Agent Trust](https://agent-trust.camel-ai.org/) [CRAB](https://github.com/camel-ai/crab?tab=readme-ov-file)\n", + "\n", + "RESOURCES\n", + "\n", + "[News](/news) [Blog](/blog) [Docs](https://docs.camel-ai.org/) [Integrations](/integration) [Team](/team)\n", + "\n", + "Get UPdates\n", + "\n", + "Thank you! Your submission has been received!\n", + "\n", + "Oops! Something went wrong while submitting the form.\n", + "\n", + "[![](https://cdn.prod.website-files.com/6659a154491a54a40551bc78/6675648577704e5e10bcdc87_Discord.svg)](http://discord.camel-ai.org/)[![](https://cdn.prod.website-files.com/6659a154491a54a40551bc78/66756482a95ea32bb2904c70_Github.svg)](https://github.com/camel-ai/camel)[![](https://cdn.prod.website-files.com/6659a154491a54a40551bc78/6736730d49b99ccd0c366086_Linkedin.svg)](https://www.linkedin.com/company/camel-ai-org/)[![](https://cdn.prod.website-files.com/6659a154491a54a40551bc78/673673125c8feebf7af520bd_X.svg)](https://x.com/CamelAIOrg)[![](https://cdn.prod.website-files.com/6659a154491a54a40551bc78/673672ff21614854f86a459c_youtube-color_svgrepo.com.svg)](https://www.youtube.com/@CamelAI)\n", + "\n", + "Copyright Ā© 2024 Eigent AI - All Rights Reserved.\n" + ] + } + ], + "source": [ + "# Initialize the Firecrawl instance\n", + "firecrawl = Firecrawl()\n", + "\n", + "# Use the `crawl` method to retrieve content from the specified URL\n", + "firecrawl_response = firecrawl.crawl(\n", + " url=\"https://www.camel-ai.org/about\" # Target URL to crawl for content\n", + ")\n", + "\n", + "print(firecrawl_response[\"status\"])\n", + "\n", + "# Print the markdown content from the first page in the crawled data\n", + "print(firecrawl_response[\"data\"][0][\"markdown\"])\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "PfXRfs2OEPtZ" + }, + "source": [ + "Step 4: Interact with CAMEL agent" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "cgXZ2dA9EPtZ", + "outputId": "86bce1b8-24ac-4eb1-e647-5fa75241a647" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CAMEL is an open-source library designed for the study of autonomous and communicative agents. The primary goal of CAMEL is to explore the scaling laws of these agents, which involves understanding their behaviors, capabilities, and potential risks when studied on a large scale. The library facilitates research by providing various types of agents, tasks, prompts, models, and simulated environments.\n", + "\n", + "CAMEL encourages collaboration and contributions from the community, inviting researchers and developers to join their efforts. They have established communication channels through platforms like Slack and Discord for interested contributors to connect and collaborate.\n", + "\n", + "The project is supported by a team of contributors and advisors, including notable figures in the field of artificial intelligence and machine learning. CAMEL aims to advance the understanding of multi-agent systems and their applications in various domains.\n" + ] + } + ], + "source": [ + "from camel.agents import ChatAgent\n", + "\n", + "# Initialize a ChatAgent\n", + "agent = ChatAgent(\n", + " system_message=\"You're a helpful assistant\", # Define the agent's role or purpose\n", + ")\n", + "\n", + "# Use the ChatAgent to generate a response based on the Firecrawl crawl data\n", + "response = agent.step(f\"Based on {firecrawl_response}, explain what CAMEL is.\")\n", + "\n", + "# Print the content of the first message in the response, which contains the assistant's answer\n", + "print(response.msgs[0].content)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "pt0IEH0jerxT" + }, + "source": [ + "## šŸ”„ Firecrawl: To Scrape\n", + "\n", + "\n", + "Scrape: This feature allows you to extract content from a single URL and convert it into various formats optimized for LLMs. The data is delivered in markdown, structured data (via LLM Extract), screenshots, or raw HTML, making it versatile for analysis and integration with other AI applications.\n", + "\n", + "\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "67X6W_KXd6Ko", + "outputId": "48241771-7b23-4748-bde8-8117b4d5e322" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'company_mission': 'We are finding the scaling law of agent', 'is_open_source': True}\n" + ] + } + ], + "source": [ + "# Define the schema\n", + "class ExtractSchema(BaseModel):\n", + " company_mission: str\n", + " is_open_source: bool\n", + "\n", + "# Perform the structured scrape\n", + "response = firecrawl.structured_scrape(\n", + " url='https://www.camel-ai.org/about', # URL to scrape data from\n", + " response_format=ExtractSchema\n", + ")\n", + "\n", + "print(response)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "fC8PS9ncTync" + }, + "source": [ + "Let's have a look how the assistant CAMEL agent can answer our questions with the response from Firecrawl." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "HyVreek1TZXQ", + "outputId": "96c62d05-33fe-4346-ed0a-5821747474a0" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The company mission of CAMEL is to find the scaling law of agents. This involves researching and understanding how the performance and behavior of autonomous and communicative agents change as they are scaled up in size, complexity, or number. The scaling law refers to the relationship between the size of the agents (or the systems they operate within) and their capabilities, efficiency, or effectiveness.\n", + "\n", + "By focusing on this mission, CAMEL aims to uncover valuable insights into how agents function in various environments and tasks, which can lead to improvements in the design and implementation of artificial intelligence systems. This research is crucial for developing more advanced and capable AI agents that can operate effectively in real-world applications.\n", + "\n", + "Additionally, CAMEL's commitment to being open-source indicates that they prioritize collaboration and transparency, allowing researchers and developers to contribute to and benefit from their findings in the field of artificial intelligence.\n" + ] + } + ], + "source": [ + "# Use the ChatAgent to generate a response based on the Firecrawl crawl data\n", + "response = agent.step(f\"Based on {response}, explain what the company mission of CAMEL is.\")\n", + "\n", + "# Print the content of the first message in the response, which contains the assistant's answer\n", + "print(response.msgs[0].content)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-_pnKO58evRN" + }, + "source": [ + "## šŸ”„ Firecrawl: To Map\n", + "\n", + "Map: This feature takes a website as input and rapidly retrieves all associated URLs, providing a quick and comprehensive overview of the siteā€™s structure. This high-speed mapping is ideal for efficient content discovery and organization." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "c_Qjpb63d8F8", + "outputId": "c969e797-fb37-4d60-a2c1-b806e5504a69" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'success': True, 'links': ['https://www.camel-ai.org', 'https://www.camel-ai.org/blog', 'https://www.camel-ai.org/checkout', 'https://www.camel-ai.org/news', 'https://www.camel-ai.org/order-confirmation', 'https://www.camel-ai.org/paypal-checkout', 'https://www.camel-ai.org/team', 'https://www.camel-ai.org/integration', 'https://www.camel-ai.org/search', 'https://www.camel-ai.org/about', 'https://www.camel-ai.org/post/crab', 'https://www.camel-ai.org/post/releasenotes-sprint4', 'https://www.camel-ai.org/post/releasenotes-sprint56', 'https://www.camel-ai.org/post/tool-usage', 'https://www.camel-ai.org/post/releasenotes-9', 'https://www.camel-ai.org/post/release-notes-10', 'https://www.camel-ai.org/post/realise-notes-9', 'https://www.camel-ai.org/post/realise-notes-8', 'https://www.camel-ai.org/post/getting-started-with-camel-ai', 'https://www.camel-ai.org/post/camel-release-notes-sprint-13', 'https://www.camel-ai.org/post/camel-release-notes-sprint-7', 'https://www.camel-ai.org/post/camel-release-notes-sprint-11', 'https://www.camel-ai.org/post/camel-release-notes-sprint-12', 'https://www.camel-ai.org/post/get-started-with-retrieval-augmented-generation', 'https://www.camel-ai.org/post/get-started-with-models---camel-101', 'https://www.camel-ai.org/post/week-4-of-may-camel-ai-release-notes', 'https://www.camel-ai.org/post/week-1-of-june-camel-ai-release-notes', 'https://www.camel-ai.org/post/3-ways-to-supercharge-camel-agents-with-web-data-using-firecrawl', 'https://www.camel-ai.org/post/mission-at-camel-ai-org-finding-the-scaling-laws-of-agents']}\n" + ] + } + ], + "source": [ + "# Call the `map_site` function from Firecrawl to retrieve all URLs from the specified website\n", + "map_result = firecrawl.map_site(\n", + " url=\"https://www.camel-ai.org\" # Target URL to map\n", + ")\n", + "\n", + "# Print the resulting map, which should contain all associated URLs from the website\n", + "print(map_result)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "92p51qPOVDpC", + "outputId": "9238d8c5-000e-4633-af4f-db4742649d31" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The main website for CAMEL-AI is [https://www.camel-ai.org](https://www.camel-ai.org).\n" + ] + } + ], + "source": [ + "# Use the ChatAgent to generate a response based on the Firecrawl crawl data\n", + "response = agent.step(f\"Based on {map_result}, which one is the main website for CAMEL-AI.\")\n", + "\n", + "# Print the content of the first message in the response, which contains the assistant's answer\n", + "print(response.msgs[0].content)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "YMRyjmWuU5a_" + }, + "source": [ + "## Conclusion\n", + "\n", + "\n", + "In conclusion, using Firecrawl within the CAMEL framework streamlines the process of web data extraction and enhances your agents capabilities. With Firecrawlā€™s powerful features like Scrape, Crawl, and Map, you can efficiently gather content in formats ready for LLMs to use, directly feeding into CAMEL-AIā€™s multi-agent workflows. This setup not only simplifies data collection but also enables more intelligent and insightful agents." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "XVjsJvXQy3V8" + }, + "source": [ + "ā­ **Star the Repo**\n", + "\n", + "If you find CAMEL useful or interesting, please consider giving it a star on [GitHub](https://github.com/camel-ai/camel)! Your stars help others find this project and motivate us to continue improving it." + ] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +}