When building the platform, add the specs for each connector in the seed

[cgardens]

Tell us about the problem you're trying to solve

We are trying to add the spec to the source/destination definitions table so that we do not need to run the spec method on a connector every time we need a spec. See the epic for more info: #6174.

Since we want to get to the place where spec is a required column in the connector definitions table, we need to figure out a way at build time to guarantee that all of the specs are loaded in the seed.

Describe the solution you’d like

For any connector that Airbyte publishes, we already push its spec into this GCS bucket. We could continue to leverage it in the build. For each connector in the seed, fetch its spec from the the bucket (if it is not already in the seed).

If the spec is not in the cache, we can probably just fail. The alternative is that we could attempt to run the spec method for that container as part of the build. Since we have full control over the bucket spec, it makes sense to use it then adding another, potentially brittle code path.

As part of this project, we will need to store the spec of each connector in the source code. We want to keep the connector definitions human readable, so the specs should be stored separately in the resources directory.

What you'll learn

• How we seed connectors into the Airbyte OSS platform
• Gradle!