Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add dashscope multimodal #92

Merged
merged 19 commits into from
Apr 9, 2024
Merged

add dashscope multimodal #92

Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
8fd75b7
add dashscope multimodal
qbc2016 Mar 20, 2024
9c4bd5f
add more note
qbc2016 Mar 21, 2024
a75833a
fix unittest
qbc2016 Mar 21, 2024
d6365be
fix typo
qbc2016 Mar 25, 2024
c0f747d
merge main
qbc2016 Mar 26, 2024
d44f6c8
add format in dashscope multimodal
qbc2016 Mar 27, 2024
9eacb46
merge main and fix conflicts
qbc2016 Apr 3, 2024
3eda2c0
minor changes
qbc2016 Apr 3, 2024
a87c490
upt docs
DavdGao Apr 4, 2024
4e0e78a
Merge remote-tracking branch 'modelscope/main' into dev_dash_multi
DavdGao Apr 4, 2024
be5abe1
Update tutorial for dashscope multimodal api; unify the prompt constr…
DavdGao Apr 4, 2024
a47a60d
reformat
DavdGao Apr 5, 2024
ffe2ecb
Merge remote-tracking branch 'modelscope/main' into dev_dash_multi
DavdGao Apr 8, 2024
b3373a8
modify "multi modal conversation" into "multimodal"
DavdGao Apr 8, 2024
947f129
adjust the order of text and multimodal urls; add api link in docstri…
DavdGao Apr 8, 2024
9a80615
modified accordingly
DavdGao Apr 8, 2024
2685636
modified accordingly
DavdGao Apr 9, 2024
4a13c3d
clean http url
DavdGao Apr 9, 2024
c42d936
change token counting strategy
DavdGao Apr 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 15 additions & 14 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,20 +65,21 @@ applications in a centralized programming manner for streamlined development.
AgentScope provides a list of `ModelWrapper` to support both local model
services and third-party model APIs.

| API | Task | Model Wrapper | Example Configuration | Some Supported Models |
|------------------------|-----------------|---------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|-------------------------------------------|
| OpenAI API | Chat | [`OpenAIChatWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/openai_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#openai-api) | gpt-4, gpt-3.5-turbo, ... |
| | Embedding | [`OpenAIEmbeddingWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/openai_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#openai-api) | text-embedding-ada-002, ... |
| | DALL·E | [`OpenAIDALLEWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/openai_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#openai-api) | dall-e-2, dall-e-3 |
| DashScope API | Chat | [`DashScopeChatWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/dashscope_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#dashscope-api) | qwen-plus, qwen-max, ... |
| | Image Synthesis | [`DashScopeImageSynthesisWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/dashscope_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#dashscope-api) | wanx-v1 |
| | Text Embedding | [`DashScopeTextEmbeddingWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/dashscope_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#dashscope-api) | text-embedding-v1, text-embedding-v2, ... |
| Gemini API | Chat | [`GeminiChatWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/gemini_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#gemini-api) | gemini-pro, ... |
| | Embedding | [`GeminiEmbeddingWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/gemini_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#gemini-api) | models/embedding-001, ... |
| ollama | Chat | [`OllamaChatWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/ollama_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#ollama-api) | llama2, Mistral, ... |
| | Embedding | [`OllamaEmbeddingWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/ollama_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#ollama-api) | llama2, Mistral, ... |
| | Generation | [`OllamaGenerationWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/ollama_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#ollama-api) | llama2, Mistral, ... |
| Post Request based API | - | [`PostAPIModelWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/post_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#post-request-api) | - |
| API | Task | Model Wrapper | Example Configuration | Some Supported Models |
|------------------------|-----------------|---------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|----------------------------------------------|
| OpenAI API | Chat | [`OpenAIChatWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/openai_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#openai-api) | gpt-4, gpt-3.5-turbo, ... |
| | Embedding | [`OpenAIEmbeddingWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/openai_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#openai-api) | text-embedding-ada-002, ... |
| | DALL·E | [`OpenAIDALLEWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/openai_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#openai-api) | dall-e-2, dall-e-3 |
| DashScope API | Chat | [`DashScopeChatWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/dashscope_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#dashscope-api) | qwen-plus, qwen-max, ... |
| | Image Synthesis | [`DashScopeImageSynthesisWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/dashscope_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#dashscope-api) | wanx-v1 |
| | Text Embedding | [`DashScopeTextEmbeddingWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/dashscope_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#dashscope-api) | text-embedding-v1, text-embedding-v2, ... |
| | Multimodal | [`DashScopeMultiModalWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/dashscope_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#dashscope-api) | qwen-vl-v1, qwen-vl-chat-v1, qwen-audio-chat |
| Gemini API | Chat | [`GeminiChatWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/gemini_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#gemini-api) | gemini-pro, ... |
| | Embedding | [`GeminiEmbeddingWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/gemini_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#gemini-api) | models/embedding-001, ... |
| ollama | Chat | [`OllamaChatWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/ollama_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#ollama-api) | llama2, Mistral, ... |
| | Embedding | [`OllamaEmbeddingWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/ollama_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#ollama-api) | llama2, Mistral, ... |
| | Generation | [`OllamaGenerationWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/ollama_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#ollama-api) | llama2, Mistral, ... |
| Post Request based API | - | [`PostAPIModelWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/post_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#post-request-api) | - |

**Supported Local Model Deployment**

Expand Down
29 changes: 15 additions & 14 deletions README_ZH.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,20 +54,21 @@ AgentScope是一个创新的多智能体开发平台，旨在赋予开发人员

AgentScope提供了一系列`ModelWrapper`来支持本地模型服务和第三方模型API。

| API | Task | Model Wrapper | Example Configuration | Some Supported Models |
|------------------------|-----------------|---------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|-------------------------------------------|
| OpenAI API | Chat | [`OpenAIChatWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/openai_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#openai-api) | gpt-4, gpt-3.5-turbo, ... |
| | Embedding | [`OpenAIEmbeddingWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/openai_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#openai-api) | text-embedding-ada-002, ... |
| | DALL·E | [`OpenAIDALLEWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/openai_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#openai-api) | dall-e-2, dall-e-3 |
| DashScope API | Chat | [`DashScopeChatWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/dashscope_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#dashscope-api) | qwen-plus, qwen-max, ... |
| | Image Synthesis | [`DashScopeImageSynthesisWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/dashscope_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#dashscope-api) | wanx-v1 |
| | Text Embedding | [`DashScopeTextEmbeddingWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/dashscope_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#dashscope-api) | text-embedding-v1, text-embedding-v2, ... |
| Gemini API | Chat | [`GeminiChatWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/gemini_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#gemini-api) | gemini-pro, ... |
| | Embedding | [`GeminiEmbeddingWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/gemini_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#gemini-api) | models/embedding-001, ... |
| ollama | Chat | [`OllamaChatWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/ollama_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#ollama-api) | llama2, Mistral, ... |
| | Embedding | [`OllamaEmbeddingWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/ollama_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#ollama-api) | llama2, Mistral, ... |
| | Generation | [`OllamaGenerationWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/ollama_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#ollama-api) | llama2, Mistral, ... |
| Post Request based API | - | [`PostAPIModelWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/post_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#post-request-api) | - |
| API | Task | Model Wrapper | Example Configuration | Some Supported Models |
|------------------------|-----------------|---------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|----------------------------------------------|
| OpenAI API | Chat | [`OpenAIChatWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/openai_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#openai-api) | gpt-4, gpt-3.5-turbo, ... |
| | Embedding | [`OpenAIEmbeddingWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/openai_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#openai-api) | text-embedding-ada-002, ... |
| | DALL·E | [`OpenAIDALLEWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/openai_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#openai-api) | dall-e-2, dall-e-3 |
| DashScope API | Chat | [`DashScopeChatWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/dashscope_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#dashscope-api) | qwen-plus, qwen-max, ... |
| | Image Synthesis | [`DashScopeImageSynthesisWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/dashscope_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#dashscope-api) | wanx-v1 |
| | Text Embedding | [`DashScopeTextEmbeddingWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/dashscope_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#dashscope-api) | text-embedding-v1, text-embedding-v2, ... |
| | Multimodal | [`DashScopeMultiModalWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/dashscope_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#dashscope-api) | qwen-vl-v1, qwen-vl-chat-v1, qwen-audio-chat |
| Gemini API | Chat | [`GeminiChatWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/gemini_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#gemini-api) | gemini-pro, ... |
| | Embedding | [`GeminiEmbeddingWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/gemini_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#gemini-api) | models/embedding-001, ... |
| ollama | Chat | [`OllamaChatWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/ollama_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#ollama-api) | llama2, Mistral, ... |
| | Embedding | [`OllamaEmbeddingWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/ollama_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#ollama-api) | llama2, Mistral, ... |
| | Generation | [`OllamaGenerationWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/ollama_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#ollama-api) | llama2, Mistral, ... |
| Post Request based API | - | [`PostAPIModelWrapper`](https://github.com/modelscope/agentscope/blob/main/src/agentscope/models/post_model.py) | [link](https://modelscope.github.io/agentscope/en/tutorial/203-model.html#post-request-api) | - |

**支持的本地模型部署**

Expand Down
Loading
Loading