Salesforce: investigate selecting schemas/streams before the discovery step

[marcosmarxm]

Tell us about the problem you're trying to solve

Users of the Salesforce connector who have a lot of tables in their salesforce instance (1k+) find the connector difficult to use because it returns all the tables, causing the following issues:

1. discover operation is too slow when there are many tables
2. the UI is very difficult to use because there

in many cases it is unnecessary to pull all the data because the user only needs to select a few tables.

Describe the solution you’d like

as a user I would like some way of filtering the data during the config stage. Is it possible to filter by a "schema" (like in Postgres) or maybe insert a regex? I don't know what options are available from the API but would like to solve this problem somehow.

Acceptance Criteria

The output of this ticket should be a proposal of the options available to us to solve the problem above. After the proposal is reviewed by Airbyte, create tickets to implement the solution.

[htrueman]

Scoping report

1. For now we use /vXX.X/sobjects/ endpoint to obtain a list of available stream names. This endpoint supports the If-Modified-Since and If-Unmodified-Since which may be used to filter the streams by modification time. So this may be useful as in most cases users want to sync the streams which are being modified periodically.

2. BTW, the option above may be a good fit only within the narrow scope of usages, thus we may use SOSL to find out the objects needed for sync. Here is the example with similar usage.
   So that way, we'll need to add the searchTerm into the connector config.

3. For the option 2 UX we may add a list of mapped objects, so we may set the search terms and how to lookup with the given term. E.g:

   • term AI, lookup starts with;
   • term user, lookup ends with;
   • term BusinessHours, lookup exact.

[sherifnada]

It looks like it's also possible to achieve this by creating a dedicated salesforce user with specific permissions, see the example from fivetran

[VasylLazebnyk]

It looks like it's also possible to achieve this by creating a dedicated salesforce user with specific permissions, see the example from fivetran

Sherif - Yevheniy created a separate ticket for your request and I moved it to the next milestone Jan 14.