603 better oauth datasources

[NaderRNA]

Ability to connect to datasources (airbyte) using OAuth login page

Uses passport.js to authenticate the OAuth, gets the refreshToken (which is the only thing airbyte needs to establish a connection to a source with OAuth).

Aims to keep current datasource flow as unchanged as possible.

We only have client ID and client Secrets for the following which will be the only ones supported:
(crossed out oauth providers are implemented)

• Hubspot
• Salesforce (forcedotcom)
• Airtable
• Google Ads (not sure if we have an approved clientId and secret)
• Xero
• Slack

[github-actions]

File	Coverage
All files	96%	✅
src/lib/utils/validationutils.ts	95%	✅

Minimum allowed coverage is 80%

Generated by 🐒 cobertura-action against 0e14b90

[github-advanced-security]

ESLint found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

[NaderRNA]

Current status of the code:

1. Successfully authenticates the user with specifically HubSpot currently.
2. Passes the refreshToken back through the callback function (OAuth controller)
3. Callback redirects back to datasource/add page with information required to set up datasource
4. CreateDatasourceForm makes a POST with the sourceConfig set to the required config for the given oauth connector
5. That then creates the TestDatasource object using the OAuth credentials
6. Continue datasource creation as standard using the existing flow

The next few commits will likely be refactoring the implementation to do the following in order:

1. pass the datasource name and datasource description through the flow
2. Provide the user with better feedback indicating what's happening (adding a new loader for OAuth to say that authentication was successful and Agent Cloud is testing the datasource)
3. Make the entire flow more compatible for other oauth providers, not just hubspot

[NaderRNA]

I merged with #622 since locally rabbitmq was broken on my ubuntu environment, those changes fixed the issue with airbyte being unable to connect to rabbitmq as a destination locally for development. Consider removing that merge if it's unnecessary to push to dev/prod

To fix the problem, we will introduce a rate-limiting middleware using the express-rate-limit package. This middleware will limit the number of requests a client can make to the server within a specified time window. We will apply this middleware to the routes that use the useJWT middleware to ensure that authorization operations are rate-limited.

1. Install the express-rate-limit package if it is not already installed.
2. Import the express-rate-limit package in the webapp/src/router.ts file.
3. Set up a rate limiter with appropriate configuration (e.g., maximum 100 requests per 15 minutes).
4. Apply the rate limiter to the routes that use the useJWT middleware.

To fix the problem, we need to introduce rate limiting to the route handlers that perform authorization. The best way to achieve this is by using the express-rate-limit package, which allows us to easily set up and apply rate limiting middleware to the relevant routes.

1. Install the express-rate-limit package.
2. Import the express-rate-limit package in the webapp/src/router.ts file.
3. Set up a rate limiter with appropriate configuration (e.g., maximum of 100 requests per 15 minutes).
4. Apply the rate limiter to the relevant routes.

To fix the problem, we need to introduce rate limiting to the routes that use the useJWT middleware. We can use the express-rate-limit package to achieve this. The rate limiter should be configured to allow a reasonable number of requests per window of time to prevent abuse while not hindering legitimate users.

1. Install the express-rate-limit package.
2. Import the express-rate-limit package in the webapp/src/router.ts file.
3. Configure the rate limiter with appropriate settings.
4. Apply the rate limiter to the routes that use the useJWT middleware.

[NaderRNA]

Somewhere along the line here a bug was introduced that replaced the file upload/drop area with the connectors dropdown

[ragyabraham]

@NaderRNA what's happening with this PR?? where are we with this??

[NaderRNA]

Currently the branch supports custom OAuth implementation for any OAuth provider where we have a Client ID and Client Secret for the oauth. Currently HubSpot is completley functional using OAuth but there are some issues with out Airtable, Salesforce and Xero app credentials so I'm just trying to iron those out with Andrew.
If you'd like a version that can go to prod alongside BYO VectorDBs I can modify this code so it won't break prod and will work for HubSpot.
But currently if you know the OAuth provider endpoint and required scopes you can create any custom OAuth implementation using this implementation of the code, nothing is hardcoded so introduction of new OAuth implementation is a lot easier for both account creation and datasource auth.