remove config persistence from seeding logic #18749

cgardens · 2022-11-01T01:42:02Z

What

Goal: totally remove config persistence
This PR removes config persistence from the connector definition seeding logic and replaces it with config repository.

cgardens · 2022-11-01T01:42:55Z

airbyte-config/init/src/main/java/io/airbyte/config/init/ApplyDefinitionsHelper.java

@@ -49,9 +48,9 @@ public void apply(final boolean updateAll) throws JsonValidationException, IOExc
    } else {
      // todo (pedroslopez): Logic to apply definitions should be moved outside of the


@pedroslopez i think this PR does this todo. i don't have context into why it wasn't all just using loadData in the first place. is there any reason not to just reduce down to this one code path?

The reason for this updateAll branching is because there are still some differences in how Cloud applies definitions vs OSS, but maybe there's a way to override the updateActorDefinitions method for Cloud instead of using this helper class. I think ideally we eventually standardize the behaviors between OSS and Cloud.

cgardens · 2022-11-01T01:43:15Z

airbyte-db/db-lib/src/main/java/io/airbyte/db/ContextQueryFunction.java

 import java.sql.SQLException;
 import org.jooq.DSLContext;

 @FunctionalInterface
 public interface ContextQueryFunction<T> {

-  T query(DSLContext context) throws SQLException;
+  T query(DSLContext context) throws SQLException, IOException;


pedroslopez · 2022-11-01T15:19:06Z

airbyte-cron/src/main/java/io/airbyte/cron/selfhealing/DefinitionsUpdater.java

@@ -69,7 +69,7 @@ void updateDefinitions() {
          remoteDefinitionsProvider.getDestinationDefinitions().size());

      try {
-        final ApplyDefinitionsHelper applyHelper = new ApplyDefinitionsHelper(configRepository, remoteDefinitionsProvider);
+        final ApplyDefinitionsHelper applyHelper = new ApplyDefinitionsHelper(configDatabase, configRepository, remoteDefinitionsProvider);


Note that this is also used in BootloaderApp, and cloud's equivalent bootloader as well

pedroslopez · 2022-11-01T15:20:39Z

airbyte-config/init/src/main/java/io/airbyte/config/init/ApplyDefinitionsHelper.java

@@ -49,9 +48,9 @@ public void apply(final boolean updateAll) throws JsonValidationException, IOExc
    } else {
      // todo (pedroslopez): Logic to apply definitions should be moved outside of the


The reason for this updateAll branching is because there are still some differences in how Cloud applies definitions vs OSS, but maybe there's a way to override the updateActorDefinitions method for Cloud instead of using this helper class. I think ideally we eventually standardize the behaviors between OSS and Cloud.

benmoriceau · 2022-11-01T17:48:32Z

...-config/config-persistence/src/main/java/io/airbyte/config/persistence/ConfigRepository.java

-  public ConfigPersistence getConfigPersistence() {
-    return persistence;
+  @VisibleForTesting
+  ConfigRepository(final ConfigPersistence persistence, final Database database, final ActorDefinitionMigrator actorDefinitionMigrator) {


Can we keep only this constructor? Otherwise it will make us to move away from Micronaut, We should update the bean of the different BeanFactories.

cgardens · 2022-11-01T18:56:27Z

This build is broken, but it is fixed in this PR #18800 that builds off of it. Will make sure build passes before merging to master, but please don't let it block review.

* master: (59 commits) 🪟🔧 Remove styled components (round 1) (#18766) Bump Airbyte version from 0.40.17 to 0.40.18 (#18827) ci: add job and run id to test reports (#18832) hide ConfigPersistence inside ConfigRepository to discourage use (#18803) Remove the bulk actions from ConfigPersistence (#18800) remove config persistence from seeding logic (#18749) Remove ConfigPersistence usage from SecretsMigrator (#18747) Add normalization changelog and bump normalization version in platform (#18813) 🐛Source Exchange Rates: Fix handling error during check connection (#18726) 🐛Destination Google Sheets: Fix empty headers list (#18729) Bump helm chart version reference to 0.40.40 (#18815) Use equalsIgnoreCase (#18810) [charts/airbyte-cron] Cleanup env vars (#18787) 🐛 Source Facebook Marketing: reduce request limit after specific error (#18734) Parameterize test_empty_streams and test_stream_with_1_airbyte_column by destination (#18197) Correct coinmarket spec (#18790) ci: use custom test-reporter action to upload job results (#18004) Fix unit tests in source relational db (#18789) query to include data plane attributes (#18531) Mark/update notification settings design (#18159) ...

* solve conflicts * solve conflict in json schema * bump to version 0.1.8 for the changes of this pr * change ad account id in the schemas * query to include data plane attributes (#18531) * query to include data plane attributes * rename functions * fix java build * more renaming fix * Fix unit tests in source relational db (#18789) * Fix unit tests * Add extra test case for record count > 1 * Store record count in variable * ci: use custom test-reporter action to upload job results (#18004) * ci: use custom action to upload job results * Correct coinmarket spec (#18790) * correct coinmarket spec * remove duplicate support normalization from source spec * rollback coinmarketcap version in source def seed * update connector version * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Parameterize test_empty_streams and test_stream_with_1_airbyte_column by destination (#18197) * Remove lines that always add Postgres to list of destinations * Parameterize all tests in test_ephemeral by destination * 🐛 Source Facebook Marketing: reduce request limit after specific error (#18734) * 🐛 Source Facebook Marketing: reduce request limit after specific error * 🐛 Source Facebook Marketing: bump version; update docs * 🐛 Source Facebook Marketing: add test * 🐛 Source Facebook Marketing: increase timeout * [charts/airbyte-cron] Cleanup env vars (#18787) * [charts/airbyte-cron] Cleanup env vars * Remove unused env var * Use equalsIgnoreCase (#18810) * Bump helm chart version reference to 0.40.40 (#18815) Co-authored-by: perangel <perangel@users.noreply.github.com> Co-authored-by: Kyryl Skobylko <xpuska513@gmail.com> * 🐛Destination Google Sheets: Fix empty headers list (#18729) * Fix empty headers list * Updated PR number * Bumped version * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * 🐛Source Exchange Rates: Fix handling error during check connection (#18726) * Fix handling error during check connection * Updated PR number * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * Add normalization changelog and bump normalization version in platform (#18813) * Remove ConfigPersistence usage from SecretsMigrator (#18747) * remove config persistence from seeding logic (#18749) * Remove the bulk actions from ConfigPersistence (#18800) * hide ConfigPersistence inside ConfigRepository to discourage use (#18803) * ci: add job and run id to test reports (#18832) * Bump Airbyte version from 0.40.17 to 0.40.18 (#18827) Co-authored-by: grishick <grishick@users.noreply.github.com> * 🪟🔧 Remove styled components (round 1) (#18766) * refactor EditorHeader (untested) * refactor BaseClearView * delete unused Subtitle * refactor ConfirmationModal * refactor Arrow * refactor BulkHeader * refactor CatalogTreeSearch * refactor StreamFieldTable * refactor StreamHeader * refactor ConnectorIcon * refactor TreeRowWrapper * refactor DeleteBlock * refactor EmptyResourceBlock * revert unintended element change * fixed acceptance tests (#18699) * 🪟🔧 Reactor Breadcrumbs component to use anchors (#18764) * refactor breadcrumbs to use actual links * PR comments on styles * increase test timeout for some webapp tests to prevent flakes (#18807) * Remove "Filters and Segments" from Google Analytics v4 (#18508) Filters and Segments info was incorrectly added to the Google Analytics v4 connector instead of the Google Analytics (Universal Analytics) Connector. * Add notes about EU OAUth (#18835) EU OAuth is not fully tested so adding a note to account for that. * 🪟🐛 Fix: visual regression in ConnectorIcon (#18849) * fix visual regression * remove unused prop * Add links to demo page (#18828) * mysql-source:fix tinyint unsigned handling (#18619) * mysql-source:fix tinyint unsigned handling * update doc * format * upgrade version * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> * 🪟 🎉 Allow environment specific sections in docs (#18829) * Allow environment specific sections in docs * Change syntax to lower case * ci: replace GITHUB_OUTPUT with GITHUB_ENV on multiline variables (#18809) * ci: replace GITHUB_OUTPUT with GITHUB_ENV on multiline variables * ci: replace github set-ouput with new syntax in ./tools/bin/ * Add connection ID to span (#18852) * edited connector docs (#18855) * 🪟 🔧 Upgrade husky to 8.0.1 (#18719) * Upgrade Husky * Upgrade Husky * Upgrade Husky * Upgrade Husky * Upgrade Husky * ci: replace GITHUB_OUTPUT with GITHUB_ENV for multiline variables (#18857) * Avoid NPE when adding connection ID to trace (#18856) * Filter exit errors by operation name (#18850) * add label * auto-bump connector version Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Xiaohan Song <xiaohan@airbyte.io> Co-authored-by: Liren Tu <tuliren@gmail.com> Co-authored-by: Conor <cpdeethree@users.noreply.github.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Greg Solovyev <grishick@users.noreply.github.com> Co-authored-by: Artem Inzhyyants <36314070+artem1205@users.noreply.github.com> Co-authored-by: perangel <perangel@gmail.com> Co-authored-by: Jonathan Pearlin <jonathan@airbyte.io> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: perangel <perangel@users.noreply.github.com> Co-authored-by: Kyryl Skobylko <xpuska513@gmail.com> Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com> Co-authored-by: Charles <charles@airbyte.io> Co-authored-by: Octavia Squidington III <90398440+octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Joey Marshment-Howell <josephkmh@users.noreply.github.com> Co-authored-by: darynaishchenko <80129833+darynaishchenko@users.noreply.github.com> Co-authored-by: Michael Siega <109092231+mfsiega-airbyte@users.noreply.github.com> Co-authored-by: Tyler B <104733644+TBernstein4@users.noreply.github.com> Co-authored-by: Yowan Ramchoreeter <26179814+YowanR@users.noreply.github.com> Co-authored-by: Tim Roes <tim@airbyte.io> Co-authored-by: Subodh Kant Chaturvedi <subodh1810@gmail.com> Co-authored-by: Volodymyr Pochtar <v.pochtar@gmail.com> Co-authored-by: Amruta Ranade <11484018+Amruta-Ranade@users.noreply.github.com>

cgardens commented Nov 1, 2022

View reviewed changes

cgardens temporarily deployed to more-secrets November 1, 2022 01:45 Inactive

pedroslopez reviewed Nov 1, 2022

View reviewed changes

cgardens temporarily deployed to more-secrets November 1, 2022 15:30 Inactive

cgardens temporarily deployed to more-secrets November 1, 2022 16:08 Inactive

cgardens temporarily deployed to more-secrets November 1, 2022 16:12 Inactive

cgardens temporarily deployed to more-secrets November 1, 2022 16:15 Inactive

cgardens temporarily deployed to more-secrets November 1, 2022 16:18 Inactive

cgardens marked this pull request as ready for review November 1, 2022 17:42

cgardens changed the title ~~wip, remove config persistence from seeding logic~~ remove config persistence from seeding logic Nov 1, 2022

cgardens requested review from benmoriceau and jdpgrailsdev November 1, 2022 17:43

benmoriceau reviewed Nov 1, 2022

View reviewed changes

Base automatically changed from cgardens/remove_config_persistence_from_bootloader to master November 1, 2022 22:59

cgardens added 6 commits November 1, 2022 16:01

wip, remove config persistence from seeding logic

be2851f

clean up

d985c43

clean up

2e8019e

clean up

682eb3b

remove DefinitionProviderToConfigPersistenceAdapter

3719424

remove loadData

2e6407c

cgardens force-pushed the cgardens/remove-config-persistence-from-defs branch from 4db3a76 to 2e6407c Compare November 1, 2022 23:01

cgardens temporarily deployed to more-secrets November 1, 2022 23:03 Inactive

cgardens merged commit c9988c4 into master Nov 1, 2022

cgardens deleted the cgardens/remove-config-persistence-from-defs branch November 1, 2022 23:48

drewrasm pushed a commit to drewrasm/airbyte that referenced this pull request Nov 2, 2022

remove config persistence from seeding logic (airbytehq#18749)

300051d

natalyjazzviolin pushed a commit that referenced this pull request Nov 3, 2022

remove config persistence from seeding logic (#18749)

4aca66d

SebastienN15 mentioned this pull request Jan 4, 2023

Bootloader crashes when using image tags that don't follow semantic versioning #21006

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

remove config persistence from seeding logic #18749

remove config persistence from seeding logic #18749

cgardens commented Nov 1, 2022 •

edited

Loading

cgardens Nov 1, 2022

pedroslopez Nov 1, 2022

cgardens Nov 1, 2022

pedroslopez Nov 1, 2022

pedroslopez Nov 1, 2022

benmoriceau Nov 1, 2022

cgardens commented Nov 1, 2022

		@@ -49,9 +48,9 @@ public void apply(final boolean updateAll) throws JsonValidationException, IOExc
		} else {
		// todo (pedroslopez): Logic to apply definitions should be moved outside of the

remove config persistence from seeding logic #18749

remove config persistence from seeding logic #18749

Conversation

cgardens commented Nov 1, 2022 • edited Loading

What

cgardens Nov 1, 2022

Choose a reason for hiding this comment

pedroslopez Nov 1, 2022

Choose a reason for hiding this comment

cgardens Nov 1, 2022

Choose a reason for hiding this comment

pedroslopez Nov 1, 2022

Choose a reason for hiding this comment

pedroslopez Nov 1, 2022

Choose a reason for hiding this comment

benmoriceau Nov 1, 2022

Choose a reason for hiding this comment

cgardens commented Nov 1, 2022

cgardens commented Nov 1, 2022 •

edited

Loading