Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add couchbase vectore store support #2303

Merged

Conversation

prajwal-pai77
Copy link
Contributor

  • Add support for Couchbase as a vector store.
  • Update Langchain community version to 0.0.43.

@prajwal-pai77 prajwal-pai77 marked this pull request as ready for review May 3, 2024 09:55
@prajwal-pai77
Copy link
Contributor Author

prajwal-pai77 commented May 3, 2024

Hey @HenryHengZJ How can i add couchbase in here: https://github.com/FlowiseAI/FlowiseDocs/tree/main/integrations/langchain/vector-stores
What is the process to contribute there?

@HenryHengZJ
Copy link
Contributor

Hey @HenryHengZJ How can i add couchbase in here: https://github.com/FlowiseAI/FlowiseDocs/tree/main/integrations/langchain/vector-stores What is the process to contribute there?

thanks for updating the docs as well! here's an example PR

@HenryHengZJ
Copy link
Contributor

@prajwal-pai77 sorry minor conflict for the pnpm lock file as I've merged in some changes, if you dont mind updating that

@prajwal-pai77 prajwal-pai77 force-pushed the feature/Couchbase-vector-store branch from d35e7fc to 47f4ec7 Compare May 6, 2024 06:54
@prajwal-pai77
Copy link
Contributor Author

Hey @HenryHengZJ I am having trouble rebasing the pnpm-lock file. But i have managed to push it but it says alot of changes in the pnpm-lock file now, can you please check if this right?
Thanks

Copy link
Contributor

@HenryHengZJ HenryHengZJ left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @prajwal-pai77 !

@HenryHengZJ
Copy link
Contributor

Hey @HenryHengZJ I am having trouble rebasing the pnpm-lock file. But i have managed to push it but it says alot of changes in the pnpm-lock file now, can you please check if this right? Thanks

that's fine, its just the formatting of the lock file that makes the diff large enought to be viewed here

@HenryHengZJ HenryHengZJ merged commit 09569d0 into FlowiseAI:main May 6, 2024
2 checks passed
@HenryHengZJ
Copy link
Contributor

HenryHengZJ commented May 6, 2024

@prajwal-pai77 there's some problem when trying to create a new docker image for arm64/amd64:
image

do you have any idea? or had you came across similar issues?

@prajwal-pai77
Copy link
Contributor Author

@prajwal-pai77 there's some problem when trying to create a new docker image for arm64/amd64: image

do you have any idea? or had you came across similar issues?

Hey @HenryHengZJ I see that you have included cmake now in the docker file. Wanted to know if the issue still persists with the docker build. If it does please let me know what the issue.

@HenryHengZJ
Copy link
Contributor

@prajwal-pai77 there's some problem when trying to create a new docker image for arm64/amd64: image
do you have any idea? or had you came across similar issues?

Hey @HenryHengZJ I see that you have included cmake now in the docker file. Wanted to know if the issue still persists with the docker build. If it does please let me know what the issue.

@prajwal-pai77 included Cmake still doesnt work, as you can see the docker image build and push failed - https://github.com/FlowiseAI/Flowise/actions/runs/8977688757/job/24656878006. Here's the CI file - https://github.com/FlowiseAI/Flowise/blob/main/.github/workflows/docker-image.yml

We did managed to made it work once using another higher RAM and CPU machine, and it took up to 40gb of RAM, almost an hour to build that. It also increased the total app size by 500mb..

If there isn't any other workaround alternative, we might have to drop the couchbase integration until we figured out a way

@lokesh-couchbase
Copy link

Hey @HenryHengZJ , I went through this and found that the culprit is building the package using Alpine. I checked with the team and as of now we do not support Alpine + Arm64 for Couchbase SDK.
One of the ways to solve this is if we can shift to using Ubuntu instead of Alpine for the builds as ubuntu uses glibc instead of musl and is much more widely adopted.
Please let me know if something can be done from your end, and I have expedited the request internally as well for support of musl but it may take some time.

@HenryHengZJ
Copy link
Contributor

Hey @HenryHengZJ , I went through this and found that the culprit is building the package using Alpine. I checked with the team and as of now we do not support Alpine + Arm64 for Couchbase SDK. One of the ways to solve this is if we can shift to using Ubuntu instead of Alpine for the builds as ubuntu uses glibc instead of musl and is much more widely adopted. Please let me know if something can be done from your end, and I have expedited the request internally as well for support of musl but it may take some time.

thanks for figuring out the issue! did some little experiment and using ubuntu as base docker image seems a lil bit too heavy on top of all the dependencies we already have, bringing the total app size from 900mb to around 1.25gb

@lokesh-couchbase
Copy link

lokesh-couchbase commented May 8, 2024

Thanks for testing using Ubuntu. However there is really nothing much we can do right now. We still do recommend upgrading to glibc based distro as many of other products might have this as requirement. Even CentOS can be considered.

@HenryHengZJ
Copy link
Contributor

Thanks for testing using Ubuntu. However there is really nothing much we can do right now. We still do recommend upgrading to glibc based distro as many of other products might have this as requirement. Even CentOS can be considered.

yeah, we have a backlog item to work on upgrading the docker to debian based node like bullseye or bookworm, both of which are using glibc. But for now we'll have to temporarily remove the couchbase integration in order to push a new version (some urgent bugfix), until we have the new docker image verified working correctly with all the other integration nodes we have on Flowise

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants