feat: support AnalyticDB vector store #5586

lpdink · 2024-06-25T12:36:09Z

Description

AnalyticDB offers enterprise-level vector database services tailored for generative AI applications. For further information, please see here
This PR utilizes alibabacloud_gpdb20160503 and alibabacloud_tea_openapi for database connectivity. It leverages a pure HTTP client architecture, hence a Docker image is not provided.
Implemented and successfully executed AnalyticDBTest using certain AK/SK.
Due to local issues with Poetry, this PR does not directly add new dependencies to pyproject.toml. Could a maintainer kindly add the following dependencies?

poetry add alibabacloud_gpdb20160503="~3.8.0"
poetry add alibabacloud_tea_openapi="~0.3.9"

Type of Change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update, included: Dify Document
Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement
Dependency upgrade

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

Added the AnalyticdbVectorTest integration test which successfully passed using certain service credentials (AK/SK).

Suggested Checklist:

I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods
optional I have made corresponding changes to the documentation
optional I have added tests that prove my fix is effective or that my feature works
optional New and existing unit tests pass locally with my changes

api/configs/middleware/__init__.py

api/configs/middleware/vdb/analyticdb_config.py

bowenliang123 · 2024-06-25T15:11:36Z

As for the dependencies, add the required packages to the pyproject.toml, and then use dev/sync-poetry to update the poetry lockfile.

And I'm new to ADB concepts, any reference or ADB docs for alibabacloud_gpdb20160503 and alibabacloud_tea_openapi . The package name sound abnormal to me especially the exact date inside the name which may indicate an version or compatibility issue? Why we would have it?

api/poetry.lock

lpdink · 2024-06-25T15:33:16Z

@bowenliang123

As for the dependencies, add the required packages to the pyproject.toml, and then use dev/sync-poetry to update the poetry lockfile.

And I'm new to ADB concepts, any reference or ADB docs for alibabacloud_gpdb20160503 and alibabacloud_tea_openapi . The package name sound abnormal to me especially the exact date inside the name which may indicate an version or compatibility issue? Why we would have it?

Done, Thank you for your suggestion. I added the specific description and updated pyproject.toml . As for the naming of alibabacloud_gpdb20160503 , this is named according to the implementation time of the first API, but the module itself is newest and continuously maintained. The quick start of AnalyticDB also added to config, here

bowenliang123 · 2024-06-25T15:37:34Z

cc @JohnJyong

bowenliang123 · 2024-06-26T00:42:55Z

https://help.aliyun.com/zh/analyticdb-for-postgresql/user-guide/sdk-python-3

lpdink · 2024-06-28T04:08:22Z

@bowenliang123 Hey, The alibabacloud-tea has released version 0.3.7, and I've updated the poetry.lock. Now, urllib3 will no longer be downgraded. Could you please approve with this PR?

lpdink · 2024-06-28T08:05:03Z

@laipz8200 Hey, could you please review this PR?

lpdink · 2024-07-01T01:43:17Z

@JohnJyong Hey, I'm really sorry to bother you, but could you please kindly review this PR?

bowenliang123 · 2024-07-01T01:45:52Z

LGTM overall. And how about resolving the conflicts first, considering many changes in configs and docker file merged recently.

lpdink · 2024-07-09T01:30:33Z

@bowenliang123 I'm really sorry to bother you, but could you please remind the other reviewers to approve this PR?

bowenliang123 · 2024-07-09T01:39:41Z

Hi @lpdink , this PR is good for me. Feel free to ask Dify's member to review this PR, as I'm not one of them although I had some work in vdb features implementation and tests. cc @takatost @JohnJyong

* refs/heads/main: (51 commits) feat: tailwind related improvement (#6085) feat: support AnalyticDB vector store (#5586) feat:add tts-streaming config and future (#5492) Feat: add index bar to select tool panel of workflow (#6066) bump to 0.6.13 (#6078) fix: Inconsistency Between Actual and Debug Input Variables (#6055) refactor: revamp picker block (#4227) chore: remove tsne unused code (#6077) fix: relative in overflow div (#5998) chore(action): move docker login above Set up QEMU in build-push action workflow (#6073) remove clunky welcome message (#6069) feat: add request_params field to jina_reader tool (#5610) fix azure stream download (#6063) chore: hide tracing introduce detail (#6049) Address the issue of the absence of poetry in the development container. (#6036) Fix authorization header validation to handle bearer types correctly - "authorization config header is required" error (#6040) Fix/6034 get random order of categories in explore and workflow is missing in zh hant (#6043) Modify slack webhook url validation to allow workflow (#6041) (#6042) fix(configs): Update pydantic settings in config files (#6023) Fix/incorrect parameter extractor memory (#6038) ...

* refs/heads/feat/if-elif: feat: update if elif feat: tailwind related improvement (#6085) feat: support AnalyticDB vector store (#5586) # Conflicts: # web/app/components/header/account-setting/members-page/invite-modal/index.tsx # web/app/components/header/account-setting/members-page/operation/index.tsx # web/app/components/workflow/nodes/if-else/components/condition-item.tsx # web/app/components/workflow/nodes/if-else/components/condition-list.tsx

Gaocarri · 2024-07-27T01:45:14Z

@lpdink sorry to bother u, have you tried migrating pg to adb? I encountered some problems

lpdink · 2024-07-27T02:55:09Z

@lpdink sorry to bother u, have you tried migrating pg to adb? I encountered some problems

@Gaocarri I'm glad to help. Could you provide more details about the problem ?

Gaocarri · 2024-07-27T03:03:47Z

@lpdink sorry to bother u, have you tried migrating pg to adb? I encountered some problems

@Gaocarri I'm glad to help. Could you provide more details about the problem ?

thanks your help!

i create a discussion for my problem: #6720

i changed the docker/.env

DB_USERNAME=xxx
DB_PASSWORD=xxx
DB_HOST=xx
DB_PORT=xx
DB_DATABASE=xx

i can use psql to connect adb in container, but some troubles。

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. 👻 feat:rag Embedding related issue, like qdrant, weaviate, milvus, vector database. 💪 enhancement New feature or request labels Jun 25, 2024

crazywoola requested review from laipz8200 and JohnJyong June 25, 2024 13:20

bowenliang123 reviewed Jun 25, 2024

View reviewed changes

api/configs/middleware/__init__.py Outdated Show resolved Hide resolved

bowenliang123 suggested changes Jun 25, 2024

View reviewed changes

bowenliang123 reviewed Jun 25, 2024

View reviewed changes

api/poetry.lock Outdated Show resolved Hide resolved

lpdink force-pushed the main branch 3 times, most recently from 2815d74 to 90eb889 Compare June 28, 2024 04:08

lpdink force-pushed the main branch from 90eb889 to d9eecb8 Compare July 1, 2024 02:46

feat: support AnalyticDB vector store

f7823bb

lpdink force-pushed the main branch from d9eecb8 to f7823bb Compare July 9, 2024 01:29

JohnJyong approved these changes Jul 9, 2024

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Jul 9, 2024

JohnJyong merged commit 7c70eb8 into langgenius:main Jul 9, 2024
5 checks passed

takatost mentioned this pull request Jul 15, 2024

bump to 0.6.14 #6294

Merged

lpdink mentioned this pull request Sep 26, 2024

Docs: add analyticdb to vector store langgenius/dify-docs#289

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support AnalyticDB vector store #5586

feat: support AnalyticDB vector store #5586

lpdink commented Jun 25, 2024 •

edited

Loading

bowenliang123 commented Jun 25, 2024

lpdink commented Jun 25, 2024

bowenliang123 commented Jun 25, 2024

bowenliang123 commented Jun 26, 2024

lpdink commented Jun 28, 2024

lpdink commented Jun 28, 2024

lpdink commented Jul 1, 2024

bowenliang123 commented Jul 1, 2024

lpdink commented Jul 9, 2024

bowenliang123 commented Jul 9, 2024 •

edited

Loading

Gaocarri commented Jul 27, 2024

lpdink commented Jul 27, 2024

Gaocarri commented Jul 27, 2024

feat: support AnalyticDB vector store #5586

feat: support AnalyticDB vector store #5586

Conversation

lpdink commented Jun 25, 2024 • edited Loading

Description

Type of Change

How Has This Been Tested?

Suggested Checklist:

bowenliang123 commented Jun 25, 2024

lpdink commented Jun 25, 2024

bowenliang123 commented Jun 25, 2024

bowenliang123 commented Jun 26, 2024

lpdink commented Jun 28, 2024

lpdink commented Jun 28, 2024

lpdink commented Jul 1, 2024

bowenliang123 commented Jul 1, 2024

lpdink commented Jul 9, 2024

bowenliang123 commented Jul 9, 2024 • edited Loading

Gaocarri commented Jul 27, 2024

lpdink commented Jul 27, 2024

Gaocarri commented Jul 27, 2024

lpdink commented Jun 25, 2024 •

edited

Loading

bowenliang123 commented Jul 9, 2024 •

edited

Loading