Create crab framework

Co-authored-by: Yanjun-Chen <yanjunch@andrew.cmu.edu>
Co-authored-by: jaywu109 <e14051350@gs.ncku.edu.tw>
Co-authored-by: Zecheng Zhang <zecheng@alumni.stanford.edu>
Co-authored-by: 姚想 <yaoxiangfd@gmail.com>
Co-authored-by: Amayama <abcdcly@qq.com>

.github/actions/crab_install/action.yml

@@ -0,0 +1,36 @@
+name: 'crab_install'
+description: 'Setup python environment and install dependencies for Crab by poetry.'
+inputs:
+  python-version:
+    description: 'Python version.'
+    required: true
+    default: '3.10'
+runs:
+  using: "composite"
+  steps:
+    - name: Set up Python
+      uses: actions/setup-python@v3
+      with:
+        python-version: '${{ inputs.python-version }}'
+    - name: Install poetry
+      uses: abatilo/actions-poetry@v2
+    - name: Setup poetry virtual environment
+      run: |
+        poetry config virtualenvs.create true --local
+        poetry config virtualenvs.in-project true --local
+      shell: bash
+    - uses: actions/cache/restore@v3
+      id: cache-restore
+      name: Restore caches for the virtual environment based on poetry.lock
+      with:
+        path: ./.venv
+        key: venv-${{ hashFiles('poetry.lock') }}
+    - name: Install the project dependencies
+      run: poetry install -E visual-prompt -E server
+      shell: bash
+    - uses: actions/cache/save@v3
+      name: Save caches based on poetry.lock 
+      if: ${{ !steps.cache-restore.outputs.cache-hit }}
+      with:
+        path: ./.venv
+        key: venv-${{ hashFiles('poetry.lock') }}

.github/workflows/publish_release.yml

@@ -0,0 +1,50 @@
+name: Publish camel to PyPI / GitHub
+
+on:
+  push:
+    tags:
+      - "v*"
+
+  workflow_dispatch:
+
+jobs:
+  build-n-publish:
+    name: Build and publish to PyPI
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+
+    steps:
+      - uses: actions/checkout@v3
+      - name: Build and publish to pypi
+        uses: JRubics/poetry-publish@v1.17
+        with:
+          pypi_token: ${{ secrets.PYPI_API_KEY }}
+          ignore_dev_requirements: "yes"
+
+      - name: Create GitHub Release
+        id: create_release
+        uses: actions/create-release@v1
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # This token is provided by Actions, you do not need to create your own token
+        with:
+          tag_name: ${{ github.ref }}
+          release_name: ${{ github.ref }}
+          draft: false
+          prerelease: false
+
+      - name: Get Asset name
+        run: |
+          export PKG=$(ls dist/ | grep tar)
+          set -- $PKG
+          echo "name=$1" >> $GITHUB_ENV
+      - name: Upload Release Asset (sdist) to GitHub
+        id: upload-release-asset
+        uses: actions/upload-release-asset@v1
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        with:
+          upload_url: ${{ steps.create_release.outputs.upload_url }}
+          asset_path: dist/${{ env.name }}
+          asset_name: ${{ env.name }}
+          asset_content_type: application/zip

.github/workflows/pytest_package.yml

@@ -0,0 +1,18 @@
+# This workflow will install Python dependencies, run tests
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
+
+name: Pytest Crab package
+
+on: push
+
+jobs:
+  pytest:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python environment and install dependencies
+      uses: ./.github/actions/crab_install
+      with:
+        python-version: "3.10"
+    - name: Run pytest
+      run: poetry run pytest test/

.gitignore

@@ -0,0 +1,171 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+.vagrant/*
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+# docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/#use-with-ide
+.pdm.toml
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+.idea/
+
+.vscode/
+.python-version
+
+_build/
+
+# model parameter
+*.pth
+
+logs/

.pre-commit-config.yaml

@@ -0,0 +1,16 @@
+repos:
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    # Ruff version.
+    rev: v0.4.2
+    hooks:
+      # Run the linter.
+      - id: ruff
+      # Run the formatter.
+      - id: ruff-format
+  - repo: local
+    hooks:
+    - id: check-license
+      name: Check License
+      entry: python licenses/update_license.py . licenses/license_template.txt 
+      language: system
+      types: [python]

README.md

@@ -0,0 +1,49 @@
+# 🦀 Crab: Cross-platform Agent Benchmark for Multimodal Embodied Language Model Agents
+
+## Overview
+
+Crab is a framework for building LLM agent benchmark environments in a Python-centric way.
+
+#### Key Features
+
+* 🌐 Cross-platform
+  * Build agent environments in memory, hosted through a docker environment, installed a virtual machine, or as a distributed physical machine, as long as they can be accessed by Python functions.
+  * Let the agent access all the environments in the same time through a unified interface.
+* ⚙ ️Easy-to-use Configuration
+  * Add a new action by simply adding a `@action` decorator on a Python function.
+  * Deine the environment by integrating several actions together.
+* 📐 Novel Benchmarking Suite
+  * Define tasks and the corresponding evlauators in an intuitive Python-native way.
+  * Introduce a novel graph evaluator method proving fine-grained metrics.
+
+## Installation
+
+#### Prerequisites
+
+- Python 3.10 or newer
+- pip
+
+```bash
+pip install crab-framework[visual-prompt]
+```
+
+## Examples
+
+#### Run template environment with openai agent
+
+You can run the examples using the following command.
+
+```bash
+export OPENAI_API_KEY=<your api key>
+python examples/single_env.py
+python examples/multi_env.py
+```
+
+#### Run desktop environment with openai agent
+
+You can run the examples using the following command.
+
+```bash
+export OPENAI_API_KEY=<your api key>
+python examples/desktop_env.py "Open Firefox"
+```

crab/__init__.py

@@ -0,0 +1,17 @@
+# =========== Copyright 2024 @ CAMEL-AI.org. All Rights Reserved. ===========
+# Licensed under the Apache License, Version 2.0 (the “License”);
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an “AS IS” BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# =========== Copyright 2024 @ CAMEL-AI.org. All Rights Reserved. ===========
+# ruff: noqa: F403
+from .core import *
+
+__version__ = "0.1.0"