categorize source connectors based on types

[subodh1810]

Issue : #6116

[subodh1810]

I had a chat with @tuliren and he said the migration for updating the saved definitions is tricky and offered help to write the migration so that we can quickly get this across the finishing line.

[sherifnada]

I think billing concerns shouldn't go in the source_definition table (especially given that this is OSS). But this particular piece of metadata seems ok since it can have other usecases around UX. In the future we should put any such concerns in a separate table IMO.

[tuliren]

I had a chat with @tuliren and he said the migration for updating the saved definitions is tricky and offered help to write the migration so that we can quickly get this across the finishing line.

Yes. The co-existence of two migration systems makes it tricky.

Please do not merge this PR. I will work on this in the next hour.

[tuliren]

I think that we may not need a migration at all:

• The source_definitions.yaml is in airbyte-config:init, which is not a dependency for airybte-db:lib. If we don't introduce this dependency, the migration actually cannot read the YAML seed file.
• Currently all the airbyte configs are just json blobs, so table structure is changed.
• Each time the server is upgrades, we update the connector definition during server startup. We can leverage this process to add the new sourceType field to the json blob.

[tuliren]

@subodh1810, I have completed the PR that adds the sourceType in this PR to the configs persistence: #6304.

Can you hold off on merging this PR? I think it is safer to merge the two PRs together. After #6304 is reviewed, and I have confirmed that these changes won't break any migration locally, I will merge yours first, and then mine. WDYT?

[subodh1810]

Closing in favour of #6304