Merge branch 'lindorm-vdb' of github.com:AlwaysBluer/dify into lindor…

…m-vdb * 'lindorm-vdb' of github.com:AlwaysBluer/dify: (39 commits) Feat : add LLM model indicator in prompt generator (langgenius#10187) chore: enable vision support for models in OpenRouter that should have supported vision (langgenius#10191) chore : code generator preview hint (langgenius#10188) fix: webapp upload file (langgenius#10195) fix(api): replace current_user with end_user in file upload (langgenius#10194) feat(document_extractor): integrate unstructured API for PPTX extraction (langgenius#10180) fix(tools): suppress RuntimeWarnings in podcast audio generator (langgenius#10182) [fix] fix the bug that modify document name not effective (langgenius#10154) fix(workflow model): ensure consistent timestamp updating (langgenius#10172) fix: Cannot find declaration to go to CLEAN_DAY_SETTING (langgenius#10157) feat: add gpustack model provider (langgenius#10158) refactor(tools): Avoid warnings. (langgenius#10161) refactor(migration/model): update column types for workflow schema (langgenius#10160) Feat/add-remote-file-upload-api (langgenius#9906) fix: upload remote image preview (langgenius#9952) clean un-allowed special charters when doing indexing estimate (langgenius#10153) refactor(service): handle unsupported DSL version with warning (langgenius#10151) Add VESSL AI OpenAI API-compatible model provider and LLM model (langgenius#9474) feat: synchronize input/output variables in the panel with generated code by the code generator (langgenius#10150) Refined README for better reading experience. (langgenius#10143) ...
AlwaysBluer · Nov 2, 2024 · 34303e5 · 34303e5
2 parents 34c49cd + 2bd0c06
commit 34303e5
Show file tree

Hide file tree

Showing 158 changed files with 3,323 additions and 851 deletions.
diff --git a/.github/workflows/build-push.yml b/.github/workflows/build-push.yml
@@ -49,7 +49,7 @@ jobs:
           echo "PLATFORM_PAIR=${platform//\//-}" >> $GITHUB_ENV
 
       - name: Login to Docker Hub
-        uses: docker/login-action@v2
+        uses: docker/login-action@v3
         with:
           username: ${{ env.DOCKERHUB_USER }}
           password: ${{ env.DOCKERHUB_TOKEN }}
@@ -114,7 +114,7 @@ jobs:
           merge-multiple: true
 
       - name: Login to Docker Hub
-        uses: docker/login-action@v2
+        uses: docker/login-action@v3
         with:
           username: ${{ env.DOCKERHUB_USER }}
           password: ${{ env.DOCKERHUB_TOKEN }}

diff --git a/README.md b/README.md
@@ -46,6 +46,56 @@
 </p>
 
 
+## Table of Content
+0. [Quick-Start🚀](https://github.com/langgenius/dify?tab=readme-ov-file#quick-start)
+
+1. [Intro📖](https://github.com/langgenius/dify?tab=readme-ov-file#intro)
+
+2. [How to use🔧](https://github.com/langgenius/dify?tab=readme-ov-file#using-dify)
+
+3. [Stay Ahead🏃](https://github.com/langgenius/dify?tab=readme-ov-file#staying-ahead)
+
+4. [Next Steps🏹](https://github.com/langgenius/dify?tab=readme-ov-file#next-steps)
+
+5. [Contributing💪](https://github.com/langgenius/dify?tab=readme-ov-file#contributing)
+
+6. [Community and Contact🏠](https://github.com/langgenius/dify?tab=readme-ov-file#community--contact)
+
+7. [Star-History📈](https://github.com/langgenius/dify?tab=readme-ov-file#star-history)
+
+8. [Security🔒](https://github.com/langgenius/dify?tab=readme-ov-file#security-disclosure)
+
+9. [License🤝](https://github.com/langgenius/dify?tab=readme-ov-file#license)
+
+> Make sure you read through this README before you start utilizing Dify😊
+
+
+## Quick start
+The quickest way to deploy Dify locally is to run our [docker-compose.yml](https://github.com/langgenius/dify/blob/main/docker/docker-compose.yaml). Follow the instructions to start in 5 minutes.
+
+> Before installing Dify, make sure your machine meets the following minimum system requirements:
+> 
+>- CPU >= 2 Core
+>- RAM >= 4 GiB
+>- Docker and Docker Compose Installed
+</br>
+
+Run the following command in your terminal to clone the whole repo.
+```bash
+git clone https://github.com/langgenius/dify.git
+```
+After cloning,run the following command one by one.
+```bash
+cd dify
+cd docker
+cp .env.example .env
+docker compose up -d
+```
+
+After running, you can access the Dify dashboard in your browser at [http://localhost/install](http://localhost/install) and start the initialization process. You will be asked to setup an admin account.
+For more info of quick setup, check [here](https://docs.dify.ai/getting-started/install-self-hosted/docker-compose)
+
+## Intro
 Dify is an open-source LLM app development platform. Its intuitive interface combines AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to production. Here's a list of the core features:
 </br> </br>
 
@@ -79,73 +129,6 @@ Dify is an open-source LLM app development platform. Its intuitive interface com
   All of Dify's offerings come with corresponding APIs, so you could effortlessly integrate Dify into your own business logic.
 
 
-## Feature comparison
-<table style="width: 100%;">
-  <tr>
-    <th align="center">Feature</th>
-    <th align="center">Dify.AI</th>
-    <th align="center">LangChain</th>
-    <th align="center">Flowise</th>
-    <th align="center">OpenAI Assistants API</th>
-  </tr>
-  <tr>
-    <td align="center">Programming Approach</td>
-    <td align="center">API + App-oriented</td>
-    <td align="center">Python Code</td>
-    <td align="center">App-oriented</td>
-    <td align="center">API-oriented</td>
-  </tr>
-  <tr>
-    <td align="center">Supported LLMs</td>
-    <td align="center">Rich Variety</td>
-    <td align="center">Rich Variety</td>
-    <td align="center">Rich Variety</td>
-    <td align="center">OpenAI-only</td>
-  </tr>
-  <tr>
-    <td align="center">RAG Engine</td>
-    <td align="center">✅</td>
-    <td align="center">✅</td>
-    <td align="center">✅</td>
-    <td align="center">✅</td>
-  </tr>
-  <tr>
-    <td align="center">Agent</td>
-    <td align="center">✅</td>
-    <td align="center">✅</td>
-    <td align="center">❌</td>
-    <td align="center">✅</td>
-  </tr>
-  <tr>
-    <td align="center">Workflow</td>
-    <td align="center">✅</td>
-    <td align="center">❌</td>
-    <td align="center">✅</td>
-    <td align="center">❌</td>
-  </tr>
-  <tr>
-    <td align="center">Observability</td>
-    <td align="center">✅</td>
-    <td align="center">✅</td>
-    <td align="center">❌</td>
-    <td align="center">❌</td>
-  </tr>
-  <tr>
-    <td align="center">Enterprise Features (SSO/Access control)</td>
-    <td align="center">✅</td>
-    <td align="center">❌</td>
-    <td align="center">❌</td>
-    <td align="center">❌</td>
-  </tr>
-  <tr>
-    <td align="center">Local Deployment</td>
-    <td align="center">✅</td>
-    <td align="center">✅</td>
-    <td align="center">✅</td>
-    <td align="center">❌</td>
-  </tr>
-</table>
-
 ## Using Dify
 
 - **Cloud </br>**
@@ -166,29 +149,20 @@ Star Dify on GitHub and be instantly notified of new releases.
 
 ![star-us](https://github.com/langgenius/dify/assets/13230914/b823edc1-6388-4e25-ad45-2f6b187adbb4)
 
+## Next steps
 
+Go to [quick-start](https://github.com/langgenius/dify?tab=readme-ov-file#quick-start) to setup your Dify or setup by source code.
 
-## Quick start
-> Before installing Dify, make sure your machine meets the following minimum system requirements:
-> 
->- CPU >= 2 Core
->- RAM >= 4 GiB
-
-</br>
-
-The easiest way to start the Dify server is to run our [docker-compose.yml](docker/docker-compose.yaml) file. Before running the installation command, make sure that [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/) are installed on your machine:
+#### If you......
+If you forget your admin account, you can refer to this [guide](https://docs.dify.ai/getting-started/install-self-hosted/faqs#id-4.-how-to-reset-the-password-of-the-admin-account) to reset the password.
 
-```bash
-cd docker
-cp .env.example .env
-docker compose up -d
-```
+> Use docker compose up without "-d" to enable logs printing out in your terminal. This might be useful if you have encountered unknow problems when using Dify.
 
-After running, you can access the Dify dashboard in your browser at [http://localhost/install](http://localhost/install) and start the initialization process.
+If you encountered system error and would like to acquire help in Github issues, make sure you always paste logs of the error in the request to accerate the conversation. Go to [Community & contact](https://github.com/langgenius/dify?tab=readme-ov-file#community--contact) for more information.
 
-> If you'd like to contribute to Dify or do additional development, refer to our [guide to deploying from source code](https://docs.dify.ai/getting-started/install-self-hosted/local-source-code)
+> Please read the [Dify Documentation](https://docs.dify.ai/) for detailed how-to-use guidance. Most of the potential problems are explained in the doc.
 
-## Next steps
+> If you'd like to contribute to Dify or make additional development, refer to our [guide to deploying from source code](https://docs.dify.ai/getting-started/install-self-hosted/local-source-code)
 
 If you need to customize the configuration, please refer to the comments in our [.env.example](docker/.env.example) file and update the corresponding values in your `.env` file. Additionally, you might need to make adjustments to the `docker-compose.yaml` file itself, such as changing image versions, port mappings, or volume mounts, based on your specific deployment environment and requirements. After making any changes, please re-run `docker-compose up -d`. You can find the full list of available environment variables [here](https://docs.dify.ai/getting-started/install-self-hosted/environments).
 
@@ -228,6 +202,7 @@ At the same time, please consider supporting Dify by sharing it on social media
 * [GitHub Issues](https://github.com/langgenius/dify/issues). Best for: bugs you encounter using Dify.AI, and feature proposals. See our [Contribution Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md).
 * [Discord](https://discord.gg/FngNHpbcY7). Best for: sharing your applications and hanging out with the community.
 * [X(Twitter)](https://twitter.com/dify_ai). Best for: sharing your applications and hanging out with the community.
+* Make sure a log, if possible, is attached to an error reported to maximize solution efficiency.
 
 ## Star history
 

diff --git a/api/Dockerfile b/api/Dockerfile
@@ -55,7 +55,12 @@ RUN apt-get update \
     && echo "deb http://deb.debian.org/debian testing main" > /etc/apt/sources.list \
     && apt-get update \
     # For Security
-    && apt-get install -y --no-install-recommends zlib1g=1:1.3.dfsg+really1.3.1-1 expat=2.6.3-1 libldap-2.5-0=2.5.18+dfsg-3+b1 perl=5.40.0-6 libsqlite3-0=3.46.1-1 \
+    && apt-get install -y --no-install-recommends expat=2.6.3-2 libldap-2.5-0=2.5.18+dfsg-3+b1 perl=5.40.0-6 libsqlite3-0=3.46.1-1 \
+    && if [ "$(dpkg --print-architecture)" = "amd64" ]; then \
+        apt-get install -y --no-install-recommends zlib1g=1:1.3.dfsg+really1.3.1-1+b1; \
+    else \
+        apt-get install -y --no-install-recommends zlib1g=1:1.3.dfsg+really1.3.1-1; \
+    fi \
     # install a chinese font to support the use of tools like matplotlib
     && apt-get install -y fonts-noto-cjk \
     && apt-get autoremove -y \

diff --git a/api/configs/middleware/__init__.py b/api/configs/middleware/__init__.py
@@ -16,6 +16,7 @@
 from configs.middleware.storage.tencent_cos_storage_config import TencentCloudCOSStorageConfig
 from configs.middleware.storage.volcengine_tos_storage_config import VolcengineTOSStorageConfig
 from configs.middleware.vdb.analyticdb_config import AnalyticdbConfig
+from configs.middleware.vdb.baidu_vector_config import BaiduVectorDBConfig
 from configs.middleware.vdb.chroma_config import ChromaConfig
 from configs.middleware.vdb.couchbase_config import CouchbaseConfig
 from configs.middleware.vdb.elasticsearch_config import ElasticsearchConfig
@@ -261,5 +262,6 @@ class MiddlewareConfig(
     TidbOnQdrantConfig,
     LindormConfig,
     OceanBaseVectorConfig,
+    BaiduVectorDBConfig,
 ):
     pass
diff --git a/api/controllers/common/errors.py b/api/controllers/common/errors.py
@@ -0,0 +1,6 @@
+from werkzeug.exceptions import HTTPException
+
+
+class FilenameNotExistsError(HTTPException):
+    code = 400
+    description = "The specified filename does not exist."
diff --git a/api/controllers/common/helpers.py b/api/controllers/common/helpers.py
@@ -0,0 +1,58 @@
+import mimetypes
+import os
+import re
+import urllib.parse
+from uuid import uuid4
+
+import httpx
+from pydantic import BaseModel
+
+
+class FileInfo(BaseModel):
+    filename: str
+    extension: str
+    mimetype: str
+    size: int
+
+
+def guess_file_info_from_response(response: httpx.Response):
+    url = str(response.url)
+    # Try to extract filename from URL
+    parsed_url = urllib.parse.urlparse(url)
+    url_path = parsed_url.path
+    filename = os.path.basename(url_path)
+
+    # If filename couldn't be extracted, use Content-Disposition header
+    if not filename:
+        content_disposition = response.headers.get("Content-Disposition")
+        if content_disposition:
+            filename_match = re.search(r'filename="?(.+)"?', content_disposition)
+            if filename_match:
+                filename = filename_match.group(1)
+
+    # If still no filename, generate a unique one
+    if not filename:
+        unique_name = str(uuid4())
+        filename = f"{unique_name}"
+
+    # Guess MIME type from filename first, then URL
+    mimetype, _ = mimetypes.guess_type(filename)
+    if mimetype is None:
+        mimetype, _ = mimetypes.guess_type(url)
+    if mimetype is None:
+        # If guessing fails, use Content-Type from response headers
+        mimetype = response.headers.get("Content-Type", "application/octet-stream")
+
+    extension = os.path.splitext(filename)[1]
+
+    # Ensure filename has an extension
+    if not extension:
+        extension = mimetypes.guess_extension(mimetype) or ".bin"
+        filename = f"{filename}{extension}"
+
+    return FileInfo(
+        filename=filename,
+        extension=extension,
+        mimetype=mimetype,
+        size=int(response.headers.get("Content-Length", -1)),
+    )
diff --git a/api/controllers/console/__init__.py b/api/controllers/console/__init__.py
@@ -2,9 +2,21 @@
 
 from libs.external_api import ExternalApi
 
+from .files import FileApi, FilePreviewApi, FileSupportTypeApi
+from .remote_files import RemoteFileInfoApi, RemoteFileUploadApi
+
 bp = Blueprint("console", __name__, url_prefix="/console/api")
 api = ExternalApi(bp)
 
+# File
+api.add_resource(FileApi, "/files/upload")
+api.add_resource(FilePreviewApi, "/files/<uuid:file_id>/preview")
+api.add_resource(FileSupportTypeApi, "/files/support-type")
+
+# Remote files
+api.add_resource(RemoteFileInfoApi, "/remote-files/<path:url>")
+api.add_resource(RemoteFileUploadApi, "/remote-files/upload")
+
 # Import other controllers
 from . import admin, apikey, extension, feature, ping, setup, version
 
@@ -43,7 +55,6 @@
     datasets_document,
     datasets_segments,
     external,
-    file,
     hit_testing,
     website,
 )

diff --git a/api/controllers/console/apikey.py b/api/controllers/console/apikey.py
@@ -10,8 +10,7 @@
 from models.model import ApiToken, App
 
 from . import api
-from .setup import setup_required
-from .wraps import account_initialization_required
+from .wraps import account_initialization_required, setup_required
 
 api_key_fields = {
     "id": fields.String,

diff --git a/api/controllers/console/app/advanced_prompt_template.py b/api/controllers/console/app/advanced_prompt_template.py
@@ -1,8 +1,7 @@
 from flask_restful import Resource, reqparse
 
 from controllers.console import api
-from controllers.console.setup import setup_required
-from controllers.console.wraps import account_initialization_required
+from controllers.console.wraps import account_initialization_required, setup_required
 from libs.login import login_required
 from services.advanced_prompt_template_service import AdvancedPromptTemplateService
 

diff --git a/api/controllers/console/app/agent.py b/api/controllers/console/app/agent.py
@@ -2,8 +2,7 @@
 
 from controllers.console import api
 from controllers.console.app.wraps import get_app_model
-from controllers.console.setup import setup_required
-from controllers.console.wraps import account_initialization_required
+from controllers.console.wraps import account_initialization_required, setup_required
 from libs.helper import uuid_value
 from libs.login import login_required
 from models.model import AppMode

diff --git a/api/controllers/console/app/annotation.py b/api/controllers/console/app/annotation.py
@@ -6,8 +6,11 @@
 from controllers.console import api
 from controllers.console.app.error import NoFileUploadedError
 from controllers.console.datasets.error import TooManyFilesError
-from controllers.console.setup import setup_required
-from controllers.console.wraps import account_initialization_required, cloud_edition_billing_resource_check
+from controllers.console.wraps import (
+    account_initialization_required,
+    cloud_edition_billing_resource_check,
+    setup_required,
+)
 from extensions.ext_redis import redis_client
 from fields.annotation_fields import (
     annotation_fields,

diff --git a/api/controllers/console/app/app.py b/api/controllers/console/app/app.py
@@ -6,8 +6,11 @@
 
 from controllers.console import api
 from controllers.console.app.wraps import get_app_model
-from controllers.console.setup import setup_required
-from controllers.console.wraps import account_initialization_required, cloud_edition_billing_resource_check
+from controllers.console.wraps import (
+    account_initialization_required,
+    cloud_edition_billing_resource_check,
+    setup_required,
+)
 from core.ops.ops_trace_manager import OpsTraceManager
 from fields.app_fields import (
     app_detail_fields,

diff --git a/api/controllers/console/app/audio.py b/api/controllers/console/app/audio.py
@@ -18,8 +18,7 @@
     UnsupportedAudioTypeError,
 )
 from controllers.console.app.wraps import get_app_model
-from controllers.console.setup import setup_required
-from controllers.console.wraps import account_initialization_required
+from controllers.console.wraps import account_initialization_required, setup_required
 from core.errors.error import ModelCurrentlyNotSupportError, ProviderTokenNotInitError, QuotaExceededError
 from core.model_runtime.errors.invoke import InvokeError
 from libs.login import login_required

diff --git a/api/controllers/console/app/completion.py b/api/controllers/console/app/completion.py
@@ -15,8 +15,7 @@
     ProviderQuotaExceededError,
 )
 from controllers.console.app.wraps import get_app_model
-from controllers.console.setup import setup_required
-from controllers.console.wraps import account_initialization_required
+from controllers.console.wraps import account_initialization_required, setup_required
 from controllers.web.error import InvokeRateLimitError as InvokeRateLimitHttpError
 from core.app.apps.base_app_queue_manager import AppQueueManager
 from core.app.entities.app_invoke_entities import InvokeFrom