Generate seed connector specs on build #7501

lmossman · 2021-10-29T23:38:30Z

What

Resolves #7142.

The result of this PR is that during the platform build, seed connector definitions are read from the resource files, and specs for them are fetched from GCS and written to corresponding resource files, unless the specs have already been written there. The goal is that specs for all seeded connector definitions are also available in the seed and they never need to be fetched during server startup.

How

This is achieved through a gradle build script that fetches the current seed definition and spec resource files, compares them to see which specs need to be fetched, fetches the specs, and writes them to the spec resource files.

Separately, a change is made to the YamlSeedConfigPersistence class to read the spec resource files along with the definition files, and merge the specs into the definitions before returning them.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

lmossman · 2021-10-29T23:44:12Z

airbyte-config/specs/src/main/java/io/airbyte/config/specs/GcsBucketSpecFetcher.java

+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class GcsBucketSpecFetcher {


This class replaces the logic that used to live inside the GcsBucketSpecCacheSchedulerClient because this logic is also needed by the new SeedConnectorSpecGenerator script. Moving it was necessary to avoid a circular dependency.

lmossman · 2021-10-30T00:17:47Z

My only concern with these changes is that if someone is actively using an old version of a connector in the seed, then this logic may cause the spec of the most recent version of that connector to be loaded into the config database, which will cause the SpecFetcher to then return that updated spec, despite the fact that the old version of the docker container is actually being used in the connection.

Is this a valid concern?

cgardens

Very nice! A bunch of cosmetic comments but otherwise looks good.

@sherifnada do you have an opinion on whether for seeds we should store specs as a single file or multiple files?

cgardens · 2021-10-30T00:43:13Z

...ler/client/src/test/java/io/airbyte/scheduler/client/BucketSpecCacheSchedulerClientTest.java

@@ -10,37 +10,37 @@
 import static org.mockito.Mockito.verifyNoInteractions;
 import static org.mockito.Mockito.when;

+import io.airbyte.config.specs.GcsBucketSpecFetcher;


it seems like most of the tests in here should move to GcsBucketSpecFetcherTest. this test suite is mostly testing that logic. I think if you move the tests with the functionality that would make sense and then all you have left here is some mocks, sanity checking that the client uses the fetcher sensibly.

To me it actually seems like this test suite is not really testing the GcsBucketSpecFetcher logic, as it mocks out that class. These tests are checking if the calls are delegated to the next client in the chain if the call to the GCS fetcher fails.

However, your comment inspired me to write tests for GcsBucketSpecFetcher that truly test the logic in that class: https://github.com/airbytehq/airbyte/pull/7501/files#diff-7ba5d616dd88de6fdfcaaef8c81e0e68a42c28f09bf67d0de0266796f151a538R20

agreed! thank you!

cgardens · 2021-10-30T00:46:39Z

airbyte-config/specs/src/main/java/io/airbyte/config/specs/SeedConnectorSpecGenerator.java

+  static {
+    SEED_ROOT_OPTION.setRequired(true);
+    OPTIONS.addOption(SEED_ROOT_OPTION);
+  }


I think we can avoid this tatic block if you declare OPTIONS after SEED_ROOT_OPTION and do something likew:

private static final Options OPTIONS = new Options().addOption(SEED_ROOT_OPTION)

Also had to switch to using the Option builder so that I could set required to true at initialization time, but now the static block is gone!

cgardens · 2021-10-30T00:47:08Z

airbyte-config/specs/src/main/java/io/airbyte/config/specs/SeedConnectorSpecGenerator.java

+    return new DockerImageSpec().withDockerImage(dockerImage).withSpec(spec);
+  }
+
+  private static CommandLine parse(final String[] args) {


See Clis#parse to DRY this up. (just added to master): https://github.com/airbytehq/airbyte/pull/7301/files

cgardens · 2021-10-30T01:12:07Z

airbyte-config/specs/src/main/java/io/airbyte/config/specs/SeedConnectorSpecGenerator.java

+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**


good comment! mention fetching from the gcs bucket and that it will fail if it's not there and why we made that choice.

i think mentioning specifically the choice we made here to only pull from the bucket cache is worth mentioning. someone is going to come along and not understand why we did it. i think there's value in saying, since this is happening at build time, we depend on the bucket cache instead of needing to run docker containers during the build which could be slow and unwieldy. if there is a failure look in the bucket cache and figure out how to get the spec for the connector in there.

That makes sense, I have updated the comment

nice comment

cgardens · 2021-10-30T01:13:00Z

airbyte-config/specs/src/main/java/io/airbyte/config/specs/SeedConnectorSpecGenerator.java

+    seedConnectorSpecGenerator.run(outputRoot, ConnectorType.DESTINATION);
+  }
+
+  public void run(final Path seedRoot, final ConnectorType connectorType) throws IOException {


the enum is a nice touch.

Unfortunately, I just realized that I probably need to get rid of the enum. Because we want to reuse this class in the cloud build, and the seed definition files in cloud have slightly different names (source_definitions_mask.yaml vs source_definitions.yaml), so with the current code I am unable to pass the desired filenames to this class from cloud. And I'm guessing we don't want to have something like a CLOUD_SOURCE value in this enum, as it is in OSS

Actually, I'll probably keep the enum and just change the signature of this run() method to take in those two filenames separately

Here is the change: https://github.com/airbytehq/airbyte/pull/7501/files/59ad7e63d4b46e08119c81012ede079e6df9b054..81d33ee538f5b6f9561e9c446f8b09e2f6408564
not as pretty as before, but at least cloud can properly use this now

Arghhh, ignore all of my comments above. I was just looking at the wrong resources folder in cloud, the file names actually do match between both. Going to revert this change.

cgardens · 2021-10-30T01:13:32Z

airbyte-config/specs/src/main/java/io/airbyte/config/specs/SeedConnectorSpecGenerator.java

+
+  @VisibleForTesting
+  final List<DockerImageSpec> fetchUpdatedSeedSpecs(final JsonNode seedDefinitions, final JsonNode currentSeedSpecs) {
+    final List<String> seedDefinitionsDockerImages = MoreIterators.toList(seedDefinitions.elements()).stream()


by convention we generally put the .stream on the new line by itself when a stream chain is going to be broken up over multiple lines.

cgardens · 2021-10-30T01:15:21Z

airbyte-config/specs/src/main/java/io/airbyte/config/specs/SeedConnectorSpecGenerator.java

+        json -> new DockerImageSpec().withDockerImage(json.get(DOCKER_IMAGE_FIELD).asText())
+            .withSpec(Jsons.object(json.get(SPEC_FIELD), ConnectorSpecification.class))));
+
+    return seedDefinitionsDockerImages.stream()


this is a smart approach for handling filtering out old specs. very nice!

cgardens · 2021-10-30T01:16:58Z

...g/persistence/src/test/java/io/airbyte/config/persistence/YamlSeedConfigPersistenceTest.java

-    assertEquals("https://docs.airbyte.io/integrations/destinations/s3", s3Destination.getDocumentationUrl());
+    assertEquals(URI.create("https://docs.airbyte.io/integrations/destinations/s3"), s3Destination.getSpec().getDocumentationUrl());
+
+    System.out.println(mysqlSource.getSpec());


get rid of the sprint statement

cgardens · 2021-10-30T01:17:35Z

...g/persistence/src/test/java/io/airbyte/config/persistence/YamlSeedConfigPersistenceTest.java

@@ -49,14 +51,18 @@ public void testGetConfig() throws Exception {
    assertEquals(s3DestinationId, s3Destination.getDestinationDefinitionId().toString());
    assertEquals("S3", s3Destination.getName());
    assertEquals("airbyte/destination-s3", s3Destination.getDockerRepository());
-    assertEquals("https://docs.airbyte.io/integrations/destinations/s3", s3Destination.getDocumentationUrl());


lol. probably worthwhile to keep this line too. i know it is silly and duplicative but for now the system expects both.

Whoops, meant to add not replace. Thanks

lmossman · 2021-11-02T17:51:13Z

I have added a README to explain this process: https://github.com/airbytehq/airbyte/blob/lmossman/fetch-seed-specs-on-build/airbyte-config/specs/README.md
But happy to update any other documentation if necessary

lmossman · 2021-11-02T18:54:32Z

@davinchia one other question about this: Charles and I also realized that there is a need to run the SeedConnectorSpecGenerator script that this PR adds in the cloud build. I started to work on a PR for this (here), but I realized that I need the :airbyte-config:init module that this PR adds to publish a jar so that the cloud build can depend on it.

I have found a couple of PRs that seem to be related to enabling/disabling jar publishing for submodules:
#5275
#5097
I don't fully understand the state of jar publishing at this time. Could you help me understand what work needs to be done to have a jar published for :airbyte-config:init so that the cloud build can depend on it?

tuliren · 2021-11-02T20:25:47Z

what the intention of the getNewFields logic is in the DatabaseConfigPersistence class?

@lmossman, the rationale is that sometimes we want to add new fields to connector definitions. The sourceType in the source connector is an example. The getNewFields logic takes care of that. Without this update, the existing connector definitions can be invalid (i.e. missing a required field).

tuliren · 2021-11-02T20:53:39Z

My only concern with these changes is that if someone is actively using an old version of a connector in the seed, then this logic may cause the spec of the most recent version of that connector to be loaded into the config database, which will cause the SpecFetcher to then return that updated spec, despite the fact that the old version of the docker container is actually being used in the connection.

Is this a valid concern?

I think this is a valid concern. But this code block will only add the new fields to the connector definition. It will not bump the version of the connector. So I think the spec should remain the same, thus not a problem.

lmossman · 2021-11-02T22:55:26Z

After meeting with @tuliren and @cgardens, we agreed on a solution to the getNewFields issue - to ignore the spec field in that logic for now, and then after the faux major version bump we will remove the getNewFields logic entirely. Here is the commit to ignore it: f02e8a3

cgardens

awesome!

cgardens · 2021-11-03T03:24:50Z

...onfig/persistence/src/main/java/io/airbyte/config/persistence/DatabaseConfigPersistence.java

@@ -366,7 +366,14 @@ void updateConfigsFromSeed(final DSLContext ctx, final ConfigPersistence seedCon

      final ConnectorInfo connectorInfo = connectorRepositoryToIdVersionMap.get(repository);
      final JsonNode currentDefinition = connectorInfo.definition;
-      final Set<String> newFields = getNewFields(currentDefinition, latestDefinition);
+
+      // todo (lmossman) - this logic to remove the "spec" field is temporary; it is necessary to avoid


thank you. excellent explanatory comment!

cgardens · 2021-11-03T03:25:46Z

airbyte-config/specs/README.md

@@ -0,0 +1,16 @@
+# Generating Seed Connector Specs


thank you for adding the readme!

davinchia · 2021-11-03T03:39:57Z

Spoke to Lake offline on the publishing question.

lmossman · 2021-11-03T03:44:18Z

Yep, Davin was able to help me understand how the cloud repo interacts with the airbyte-server jar which contains the dependencies as well, and I was able to test this out successfully!

This reverts commit a534bb2.

* add specs module with logic to fetch specs on build * format + build and add gradle dependency for new script * check seed file for existing specs + refactor * add tests + a bit more refactoring * run gw format * update yaml config persistence to merge specs into definitions * add comment * delete secrets migration to be consistent with master * add dep * add tests for GcsBucketSpecFetcher * get rid of static block + format * DRY up parse call * add GCS details to comment * formatting + fix test * update comment * do not format seed specs files * change signature of run to allow cloud to reuse this script * run gw format * revert commits that change signature of run * fix comment typo Co-authored-by: Davin Chia <davinchia@gmail.com> * rename enum to be distinct from the enum in cloud * add missing dependencies between modules * add readme for seed connector spec generator * reword * reference readme in comment * ignore 'spec' field in newFields logic Co-authored-by: Davin Chia <davinchia@gmail.com>

This reverts commit a534bb2.

lmossman added 6 commits October 27, 2021 18:35

add specs module with logic to fetch specs on build

092ac43

format + build and add gradle dependency for new script

645c4ae

check seed file for existing specs + refactor

90a52a1

add tests + a bit more refactoring

fc4f133

run gw format

9e27fab

update yaml config persistence to merge specs into definitions

79365e0

github-actions bot added area/platform issues related to the platform area/scheduler labels Oct 29, 2021

lmossman commented Oct 29, 2021

View reviewed changes

add comment

7f8d6d3

lmossman temporarily deployed to more-secrets October 29, 2021 23:49 Inactive

merge with master

6fc04d7

lmossman temporarily deployed to more-secrets October 29, 2021 23:57 Inactive

delete secrets migration to be consistent with master

e868c78

lmossman temporarily deployed to more-secrets October 30, 2021 00:16 Inactive

lmossman requested a review from cgardens October 30, 2021 00:28

add dep

943eb25

github-actions bot added the area/server label Oct 30, 2021

lmossman temporarily deployed to more-secrets October 30, 2021 00:32 Inactive

cgardens requested review from airbyte-jenny and davinchia October 30, 2021 00:33

cgardens approved these changes Oct 30, 2021

View reviewed changes

add tests for GcsBucketSpecFetcher

805c818

lmossman temporarily deployed to more-secrets November 1, 2021 17:50 Inactive

get rid of static block + format

007b112

lmossman temporarily deployed to more-secrets November 1, 2021 17:57 Inactive

lmossman added 3 commits November 1, 2021 11:03

Merge branch 'master' into lmossman/fetch-seed-specs-on-build

53ab6ed

DRY up parse call

9bb59f9

add GCS details to comment

30016e8

github-actions bot added the area/protocol label Nov 2, 2021

lmossman temporarily deployed to more-secrets November 2, 2021 17:34 Inactive

add readme for seed connector spec generator

6cec994

lmossman temporarily deployed to more-secrets November 2, 2021 17:51 Inactive

reword

b81e0fa

lmossman temporarily deployed to more-secrets November 2, 2021 17:54 Inactive

reference readme in comment

a8ceffb

lmossman temporarily deployed to more-secrets November 2, 2021 17:57 Inactive

ignore 'spec' field in newFields logic

f02e8a3

lmossman temporarily deployed to more-secrets November 2, 2021 22:55 Inactive

cgardens approved these changes Nov 3, 2021

View reviewed changes

lmossman temporarily deployed to more-secrets November 3, 2021 03:36 Inactive

lmossman merged commit a534bb2 into master Nov 3, 2021

lmossman deleted the lmossman/fetch-seed-specs-on-build branch November 3, 2021 05:03

lmossman added a commit that referenced this pull request Nov 3, 2021

Revert "Generate seed connector specs on build (#7501)"

b94ee00

This reverts commit a534bb2.

jrhizor mentioned this pull request Nov 3, 2021

Bump Airbyte version from 0.30.25-alpha to 0.30.26-alpha #7599

Merged

lmossman added a commit that referenced this pull request Nov 3, 2021

Revert "Generate seed connector specs on build (#7501)"

57cc151

This reverts commit a534bb2.

lmossman restored the lmossman/fetch-seed-specs-on-build branch November 3, 2021 22:19

lmossman temporarily deployed to more-secrets November 3, 2021 22:21 Inactive

lmossman mentioned this pull request Nov 4, 2021

REDO: Generate seed connector specs on build #7613

Merged

38 tasks

schlattk pushed a commit to schlattk/airbyte that referenced this pull request Jan 4, 2022

Revert "Generate seed connector specs on build (airbytehq#7501)"

dedc8e1

This reverts commit a534bb2.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate seed connector specs on build #7501

Generate seed connector specs on build #7501

lmossman commented Oct 29, 2021 •

edited

Loading

lmossman Oct 29, 2021

lmossman commented Oct 30, 2021 •

edited

Loading

cgardens left a comment

cgardens Oct 30, 2021

lmossman Nov 1, 2021

cgardens Nov 1, 2021

cgardens Oct 30, 2021

lmossman Nov 1, 2021

cgardens Oct 30, 2021

cgardens Oct 30, 2021

lmossman Nov 1, 2021

cgardens Nov 1, 2021

lmossman Nov 1, 2021

davinchia Nov 2, 2021

cgardens Oct 30, 2021

lmossman Nov 1, 2021

lmossman Nov 1, 2021

lmossman Nov 1, 2021

lmossman Nov 1, 2021

cgardens Oct 30, 2021

cgardens Oct 30, 2021

cgardens Oct 30, 2021

cgardens Oct 30, 2021

lmossman Nov 1, 2021

lmossman commented Nov 2, 2021 •

edited

Loading

lmossman commented Nov 2, 2021

tuliren commented Nov 2, 2021

tuliren commented Nov 2, 2021 •

edited

Loading

lmossman commented Nov 2, 2021

cgardens left a comment

cgardens Nov 3, 2021

cgardens Nov 3, 2021

davinchia commented Nov 3, 2021

lmossman commented Nov 3, 2021

Generate seed connector specs on build #7501

Generate seed connector specs on build #7501

Conversation

lmossman commented Oct 29, 2021 • edited Loading

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Choose a reason for hiding this comment

lmossman commented Oct 30, 2021 • edited Loading

cgardens left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lmossman commented Nov 2, 2021 • edited Loading

lmossman commented Nov 2, 2021

tuliren commented Nov 2, 2021

tuliren commented Nov 2, 2021 • edited Loading

lmossman commented Nov 2, 2021

cgardens left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davinchia commented Nov 3, 2021

lmossman commented Nov 3, 2021

lmossman commented Oct 29, 2021 •

edited

Loading

lmossman commented Oct 30, 2021 •

edited

Loading

lmossman commented Nov 2, 2021 •

edited

Loading

tuliren commented Nov 2, 2021 •

edited

Loading