601 allow user defined vector dbs as a destination #623

iandjx · 2024-10-30T06:48:42Z

No description provided.

…connector

…ues to string before pushing to vector store

…ring that rows are sub-chunked correctly. Text is cleaned up

…ll text. set the rabbitmq host in the webapp to use the correct env variable

…-dbs-as-a-destination

…pes using serde_json::Value

Builtin support for relationships can make it easy for us to fetch which datasource objects use which vectordb

Add Mongoose Schema for Vectordb and Datasource Models

Develop

… introduced update creds function for vectordatabase trait that allows us to update api keys for qdrant and pinecone

…tination

…ttps://github.com/rnadigital/agentcloud into 601-allow-user-defined-vector-dbs-as-a-destination

webapp/src/components/vectordbs/VectorDbForm.tsx

+
+					if (!callback) {
+						fetchVectorDbFormData?.();
+						router.push(`/${resourceSlug}/vectordbs`);


To fix the problem, we need to ensure that the resourceSlug value is validated against a list of authorized redirects before using it in the URL redirection. This can be achieved by maintaining a list of allowed resourceSlug values and checking if the provided resourceSlug is in that list before performing the redirection.

Create a list of authorized resourceSlug values.

Check if the resourceSlug value is in the list of authorized values before using it in the URL redirection.

If the resourceSlug is not authorized, handle the error appropriately (e.g., show an error message or redirect to a default safe page).

webapp/src/components/vectordbs/VectorDbForm.tsx

+					toast.success('VectorDb Added');
+					if (!callback) {
+						fetchVectorDbFormData?.();
+						router.push(`/${resourceSlug}/vectordbs`);


To fix the problem, we need to ensure that the resourceSlug value is validated against a list of authorized redirects before using it in the URL redirection. This can be achieved by maintaining a list of valid resourceSlug values and checking if the provided resourceSlug is in that list before performing the redirection.

Create a list of authorized resourceSlug values.

Check if the resourceSlug value is in the list of authorized values before using it in the URL redirection.

If the resourceSlug is not authorized, handle the error appropriately (e.g., show an error message or redirect to a default page).

webapp/src/components/vectordbs/VectorDbTable.tsx

+						{vectorDbs.map(vectorDb => (
+							<tr
+								key={vectorDb._id}
+								onClick={() => router.push(`/${resourceSlug}/vectordb/${vectorDb._id}/edit`)}


To fix the problem, we should avoid using user input directly in the URL redirection. Instead, we can maintain a list of authorized redirects and validate the resourceSlug against this list before constructing the URL. This ensures that only safe, predefined values are used in the redirection.

Create a list of authorized resourceSlug values.

Validate the resourceSlug against this list before using it in the URL construction.

If the resourceSlug is not in the list, handle the error appropriately (e.g., redirect to a default page or show an error message).

webapp/src/components/vectordbs/VectorDbTable.tsx

webapp/src/router.ts

-	server.get('/register', unauthedMiddlewareChain, checkSessionWelcome, renderStaticPage(app, '/register'));
+	server.get(
+		'/login',
+		unauthedMiddlewareChain,


webapp/src/router.ts

@@ -717,6 +725,32 @@
 		variableController.deleteVariableApi
 	);

+	teamRouter.get('/vectordbs.json', vectorDbController.vectorDbsJson);
+
+	teamRouter.get('/vectordb/:vectorDbId([a-f0-9]{24}).json', vectorDbController.vectorDbJson);


webapp/src/router.ts

+	teamRouter.delete(
+		'/forms/vectordb/:vectorDbId',
+		hasPerms.one(Permissions.DELETE_VECTOR_DB),
+		vectorDbController.deleteVectorDbApi


To fix the problem, we need to introduce rate limiting to the route handler that performs the database access. We can use the express-rate-limit package to achieve this. The rate limiter will be configured to allow a certain number of requests per window of time (e.g., 100 requests per 15 minutes) and will be applied to the specific route handler.

Install the express-rate-limit package if it is not already installed.

Import the express-rate-limit package in the webapp/src/router.ts file.

Configure the rate limiter with appropriate settings.

Apply the rate limiter to the deleteVectorDbApi route handler.

…s going on

…ore loging to make the process clearer

webapp/src/db/vectordb.ts

…stination

ragyabraham and others added 30 commits September 17, 2024 12:03

getting the metadata to be in a format that complies with n8n qdrant …

1f54136

…connector

Merge branch 'master' into n8n

3d32850

Merge branch 'develop' into n8n

bf2e780

usign serde_json::to_string() rather than .to_string() to convert val…

ee56738

…ues to string before pushing to vector store

Merge branch 'master' into n8n-airbyte-upgrade

5e4a329

setting rabbitmq to listen on all addresses

db16855

setting the rabbitmq host as the local host ip

1150191

setting the rabbitmq host as the local host ip

99a02c3

retaining value types by using serde Values rathen than strings. ensu…

c5a033a

…ring that rows are sub-chunked correctly. Text is cleaned up

adding host-gateway as an extra host in the docker-compose file

2b83ce4

setting content as top line var in metadata

ba5e887

removing unwanted characters from final text

65a1c35

added clean_text fn that can be used to removed unwanted charc from a…

03b534c

…ll text. set the rabbitmq host in the webapp to use the correct env variable

using regex to match unwanted chars rather than replace

6bc4f8e

Merge branch 'n8n-airbyte-upgrade' into 601-allow-user-defined-vector…

3dd73a8

…-dbs-as-a-destination

cleaning up the payload that goes into pinecone by retaining value ty…

2747ea7

…pes using serde_json::Value

moved helpers to their own module

2e5e415

removed unused import

7cc3935

Setup mongoose

3dfdfad

Builtin support for relationships can make it easy for us to fetch which datasource objects use which vectordb

Add mongoose schema for vectordb and datasource

ec82f42

Use debug library for mongoose

fb3b0c5

Set url as optional

d921616

Merge pull request #607 from rnadigital/601-mongo-db-changes

a5939fb

Add Mongoose Schema for Vectordb and Datasource Models

Merge pull request #616 from rnadigital/develop

4b98e7a

Develop

added struct and mongo query to fetch vector db config

902ae97

dramatically reduced the number of db calls when processing messages.…

d317aa6

… introduced update creds function for vectordatabase trait that allows us to update api keys for qdrant and pinecone

Merge branch 'master' into 601-allow-user-defined-vector-dbs-as-a-des…

372c855

…tination

Add vectordb endpoints

e86b5b8

Merge branch '601-allow-user-defined-vector-dbs-as-a-destination' of h…

ddfe11f

…ttps://github.com/rnadigital/agentcloud into 601-allow-user-defined-vector-dbs-as-a-destination

basic hotfix for an issue affecting all apps on prod

3957487

github-advanced-security bot found potential problems Oct 30, 2024

View reviewed changes

iandjx and others added 17 commits October 30, 2024 15:02

Temporarily disable mongoose

8badb45

adding try-catch and some loggin in the setup of airbyte to see what'…

7873e6c

…s going on

Restore mongoose

ac479b6

adding in byo vector login in handle_embedding function

044fbb2

made check_byo_vector a helper function that can be re-used

7659056

adding a if let Ok to the check byo vector db

6875edc

adding check to ensure vector db id is present

16ceb2d

added some logging

02b9339

Set url field and namespace as not required

b975dfa

made mongo model for vector dbs match document in collection. added m…

aee174f

…ore loging to make the process clearer

adding all capabilities required to allow byo vector databases

b5e5a4b

removed useless file

5094f71

setting the collection name as the index name to create in pionecone

173fd61

coloning value that has been moved

6507214

Properly pass args to vectorstore_factory

7a41c41

Validate if byo vector db credentials are valid

eb47cc5

Use datasource id as namespace

d6d4ea4

github-advanced-security bot found potential problems Nov 7, 2024

View reviewed changes

webapp/src/db/vectordb.ts Fixed Show fixed Hide fixed

ragyabraham and others added 9 commits November 7, 2024 15:55

slight change to how we build the qdrant client

a08b73e

Remove :6334 when using qdrant

c315fc2

Remove collection name when using a qdrant vectordb

b639024

Remove router push to /variable

9ad37ef

Add rule that qdrant url should end with :6334

f0e6ec2

Add pinecone connect

8380ef2

Fix error message even if delete is successful

4307e31

Add tooltip to index field

73d1897

Merge branch 'develop' into 601-allow-user-defined-vector-dbs-as-a-de…

24f3c1f

…stination

ragyabraham merged commit 16cf333 into develop Nov 11, 2024
7 checks passed

ragyabraham deleted the 601-allow-user-defined-vector-dbs-as-a-destination branch November 11, 2024 22:35

@@ -35,2 +35,3 @@
             	const { resourceSlug } = router.query;
+            	const authorizedResourceSlugs = ['allowedSlug1', 'allowedSlug2', 'allowedSlug3']; // Add your authorized slugs here
@@ -65,3 +66,7 @@
             						fetchVectorDbFormData?.();
-            						router.push(`/${resourceSlug}/vectordbs`);
+            						if (authorizedResourceSlugs.includes(resourceSlug)) {
+            							router.push(`/${resourceSlug}/vectordbs`);
+            						} else {
+            							toast.error('Unauthorized resource slug');
+            						}
             					}

@@ -35,2 +35,3 @@
             	const { resourceSlug } = router.query;
+            	const authorizedResourceSlugs = ['validSlug1', 'validSlug2', 'validSlug3']; // Add your authorized slugs here
@@ -81,3 +82,7 @@
             						fetchVectorDbFormData?.();
-            						router.push(`/${resourceSlug}/vectordbs`);
+            						if (authorizedResourceSlugs.includes(resourceSlug)) {
+            							router.push(`/${resourceSlug}/vectordbs`);
+            						} else {
+            							toast.error('Invalid resource slug');
+            						}
             					}

@@ -24,2 +24,4 @@
             	const { resourceSlug } = router.query;
+            	const authorizedResourceSlugs = ['authorizedSlug1', 'authorizedSlug2']; // Add all authorized slugs here
+            	const isValidResourceSlug = authorizedResourceSlugs.includes(resourceSlug);
             	const [deletingVectorDb, setDeletingVectorDb] = useState(null);
@@ -121,3 +123,10 @@
             								key={vectorDb._id}
-            								onClick={() => router.push(`/${resourceSlug}/vectordb/${vectorDb._id}/edit`)}
+            								onClick={() => {
+            									if (isValidResourceSlug) {
+            										router.push(`/${resourceSlug}/vectordb/${vectorDb._id}/edit`);
+            									} else {
+            										// Handle invalid resourceSlug, e.g., redirect to a default page or show an error
+            										router.push('/defaultPage');
+            									}
+            								}}
             								className={cn(

@@ -29,2 +29,3 @@
             import fileUpload from 'express-fileupload';
+            import RateLimit from 'express-rate-limit';
             import Permissions from 'permissions/permissions';
@@ -55,2 +56,7 @@
+            const limiter = RateLimit({
+            	windowMs: 15 * 60 * 1000, // 15 minutes
+            	max: 100, // max 100 requests per windowMs
+            });
             export default function router(server, app) {
@@ -751,2 +757,3 @@
             		hasPerms.one(Permissions.DELETE_VECTOR_DB),
+            		limiter,
             		vectorDbController.deleteVectorDbApi

@@ -123,3 +123,4 @@
                 "tsconfig-paths": "^4.2.0",
-                "uuid": "^9.0.1"
+                "uuid": "^9.0.1",
+                "express-rate-limit": "^7.4.1"
               },

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

601 allow user defined vector dbs as a destination #623

601 allow user defined vector dbs as a destination #623

iandjx commented Oct 30, 2024

Provide additional feedback

Please help us improve GitHub Copilot by sharing more details about this comment.

Provide additional feedback

Please help us improve GitHub Copilot by sharing more details about this comment.

Provide additional feedback

Please help us improve GitHub Copilot by sharing more details about this comment.

Provide additional feedback

Please help us improve GitHub Copilot by sharing more details about this comment.

Package	Version	Security advisories
express-rate-limit (npm)	7.4.1	None

601 allow user defined vector dbs as a destination #623

601 allow user defined vector dbs as a destination #623

Conversation

iandjx commented Oct 30, 2024