Add a config for instance wide oauth parameters #5761

ChristopheDuong · 2021-08-31T15:23:46Z

What

Closes #5637
Closes #5638

How

Describe the solution

Introduce new airbyte configs in the ConfigSchema enum: SourceOAuthParameter & DestinationOAuthParameter to be persisted in the ConfigRepository
These configs are retrieved by the Scheduler through the DefaultSyncJobFactory that reconstructs all the configuration data necessary for a sync. When OAuth parameters are necessary for certain connectors, they are injected into the connector configurations before being scheduled/submitted to workers.
Additionally, this PR also cleans up web backend endpoints that are now unused (source & destination recreate)

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions
Connector version bumped like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

airbyte-api/src/main/openapi/config.yaml

ChristopheDuong · 2021-09-02T13:21:26Z

...sistence/src/main/java/io/airbyte/scheduler/persistence/job_factory/OAuthConfigSupplier.java

+
+  private static void injectJsonNode(ObjectNode config, ObjectNode fromConfig) {
+    for (String key : Jsons.keys(fromConfig)) {
+      config.set(key, fromConfig.get(key));


If the OAuth parameter was already part of the connector config, should we still inject and overwrite it with the one from the instance-wide values?

or should there be a special behavior here? (adding only if does not exists yet)

hmm. Good question. I think if any of the parameters is present in the config we shouldn't add it. It may end up being integration specific. For now I think the right way to handle it is don't set anything if any of the oauth config keys are present in the config

jrhizor · 2021-09-02T17:56:40Z

Will similar injection to what is happening in DefaultSyncJobFactory be added to check/discover in this PR or a separate one?

sherifnada

same question as jared about discover/check

Also, I think having seen the impl here I think it's better if we maintain the invariant that configs are always fully formed but instead of having the full secrets they would have coordinates to the secrets in a secret store.

Also, we should either disable custom DBT transformations on cloud or just not support oauth for destinations until we do

sherifnada · 2021-09-02T18:43:44Z

airbyte-config/models/src/main/resources/types/DestinationOAuthParameter.yaml

+  - configuration
+additionalProperties: false
+properties:
+  oauthParameterId:


what's this param for? could you add a description field? if this is just a surrogate key maybe id is a better name?

Yes, it's the primary key for the OAuth parameter object.

I don't think we have any objects that define id as their primary key column though?

SourceConnection is sourceId

DestinationConnection is destinationId

StandardSync is connectionId

That's why I named it that way

sgtm let's stick with the convention 👍🏼

sherifnada · 2021-09-02T18:55:42Z

...sistence/src/main/java/io/airbyte/scheduler/persistence/job_factory/OAuthConfigSupplier.java

+
+  private static void injectJsonNode(ObjectNode config, ObjectNode fromConfig) {
+    for (String key : Jsons.keys(fromConfig)) {
+      config.set(key, fromConfig.get(key));


hmm. Good question. I think if any of the parameters is present in the config we shouldn't add it. It may end up being integration specific. For now I think the right way to handle it is don't set anything if any of the oauth config keys are present in the config

sherifnada · 2021-09-02T20:54:54Z

...stence/src/main/java/io/airbyte/scheduler/persistence/job_factory/DefaultSyncJobFactory.java

      final DestinationConnection destinationConnection = configRepository.getDestinationConnection(standardSync.getDestinationId());
-
+      destinationConnection.withConfiguration(oAuthConfigSupplier.injectDestinationOAuthParameters(


this is dangerous because the secrets could leak into custom dbt images. We should not pass this.

We're thinking about reworking how to pass the config files to custom dbt transformation containers.

So the destination_config.json (used by destination connectors) should not be forwarded to the custom dbt image (only a translated profiles.yml should, thus this file should therefore omit OAuth params)

#5091

See https://github.com/airbytehq/airbyte/pull/5761/files#r702960794

airbyte-server/src/main/java/io/airbyte/server/handlers/OAuthHandler.java

ChristopheDuong · 2021-09-06T15:01:25Z

airbyte-integrations/bases/base-normalization/entrypoint.sh

@@ -29,6 +29,8 @@ function configuredbt() {
    echo "Running: transform-config --config ${CONFIG_FILE} --integration-type ${INTEGRATION_TYPE} --out ${PROJECT_DIR}"
    # Generate a profiles.yml file for the selected destination/integration type
    transform-config --config "${CONFIG_FILE}" --integration-type "${INTEGRATION_TYPE}" --out "${PROJECT_DIR}"
+    # Remove config file as it might still contain sensitive credentials
+    rm "${CONFIG_FILE}"


Normalization image is used by custom operations to translate a destination_config.json into a profiles.yml file.

We could make sure to remove the config file once we're done translating it

I think I'm not totally clear on why this is needed since it's running a trusted image? or is this re-used for custom operations?

I think I'm not totally clear on why this is needed since it's running a trusted image? or is this re-used for custom operations?

Yes,the normalization image is trusted to generate dbt's profiles.yml (from a destination_config.json file).
The resulting profiles.yml file is then transferred to run custom DBT operations in the user's specified docker image (so the latter one never needs to read/parse a destination_config.json)

SGTM - could you add a source code comment explaining why that file containing sensitive credentials is a bad thing and therefore it should be removed?

sherifnada

I think this LGTM but would like @jrhizor 's input on whether to put the oauth secrets in the jobs DB directly vs. reading them at the moment we're about to run a connector

sherifnada · 2021-09-07T21:26:29Z

airbyte-config/models/src/main/resources/types/DestinationOAuthParameter.yaml

+  - configuration
+additionalProperties: false
+properties:
+  oauthParameterId:


sgtm let's stick with the convention 👍🏼

sherifnada · 2021-09-07T21:27:39Z

airbyte-integrations/bases/base-normalization/entrypoint.sh

@@ -29,6 +29,8 @@ function configuredbt() {
    echo "Running: transform-config --config ${CONFIG_FILE} --integration-type ${INTEGRATION_TYPE} --out ${PROJECT_DIR}"
    # Generate a profiles.yml file for the selected destination/integration type
    transform-config --config "${CONFIG_FILE}" --integration-type "${INTEGRATION_TYPE}" --out "${PROJECT_DIR}"
+    # Remove config file as it might still contain sensitive credentials
+    rm "${CONFIG_FILE}"


I think I'm not totally clear on why this is needed since it's running a trusted image? or is this re-used for custom operations?

sherifnada · 2021-09-07T21:40:58Z

...-scheduler/persistence/src/main/java/io/airbyte/scheduler/persistence/DefaultJobCreator.java

@@ -109,15 +126,23 @@ public long createGetSpecJob(String integrationImage) throws IOException {
                                      String destinationDockerImageName,
                                      List<StandardSyncOperation> standardSyncOperations)
      throws IOException {
+    final JsonNode sourceConfiguration = oAuthConfigSupplier.injectSourceOAuthParameters(


@jrhizor is this the right place to be injecting the oauth secrets i.e: enqueue them directly in the jobs DB?

It's probably fine for now but it will change in the near future. This is what I'm working on eliminating this week, still working out the details of what it will look like.

Most likely we will push the combined version to config sent to Temporal.

sherifnada

what do you think is the best way to prevent this from leaking into DBT images? I created this PR but should we also add something in the worker? ignore if it's not answered by the other question i asked about reusing normalization entrypoint

jrhizor · 2021-09-07T22:38:19Z

...-scheduler/persistence/src/main/java/io/airbyte/scheduler/persistence/DefaultJobCreator.java

  }

  @Override
  public long createSourceCheckConnectionJob(SourceConnection source, String dockerImageName) throws IOException {
+    final JsonNode sourceConfiguration = oAuthConfigSupplier.injectSourceOAuthParameters(


The combining for get spec / check / discover can't happen in the JobCreator since the jobs are created directly in the DefaultSynchronousSchedulerClient. We're removing these misleading and unused functions in #5896

Oh ok, so I am reverting back to injecting OAuth parameters in the DefaultSyncJobFactory as well as handling it in DefaultSynchronousSchedulerClient for check/discover. (probably not needed for get spec)

jrhizor · 2021-09-07T22:40:24Z

...-scheduler/persistence/src/main/java/io/airbyte/scheduler/persistence/DefaultJobCreator.java

@@ -109,15 +126,23 @@ public long createGetSpecJob(String integrationImage) throws IOException {
                                      String destinationDockerImageName,
                                      List<StandardSyncOperation> standardSyncOperations)
      throws IOException {
+    final JsonNode sourceConfiguration = oAuthConfigSupplier.injectSourceOAuthParameters(


It's probably fine for now but it will change in the near future. This is what I'm working on eliminating this week, still working out the details of what it will look like.

Most likely we will push the combined version to config sent to Temporal.

jrhizor · 2021-09-07T22:43:47Z

airbyte-config/persistence/src/main/java/io/airbyte/config/persistence/ConfigRepository.java

@@ -211,6 +213,33 @@ public void writeStandardSyncOperation(final StandardSyncOperation standardSyncO
    return persistence.listConfigs(ConfigSchema.STANDARD_SYNC_OPERATION, StandardSyncOperation.class);
  }

+  public SourceOAuthParameter getSourceOAuthParams(final UUID SourceOAuthParameterId)


We're going to need to overwrite these calls to use the secrets persistence @airbyte-jenny

jrhizor · 2021-09-07T22:48:22Z

...sistence/src/main/java/io/airbyte/scheduler/persistence/job_factory/OAuthConfigSupplier.java

+
+  private static void injectJsonNode(ObjectNode config, ObjectNode fromConfig) {
+    for (String key : Jsons.keys(fromConfig)) {
+      if (!config.has(key) || isSecretMask(config.get(key).asText())) {


When are we inserting that the secrets mask is necessary to be overwritten?

I'm proposing to persist in the connector's configuration with injected and masked OAuth parameters in this PR: #5865

This would be done by web backend endpoints of the API when creating a connector.

The motivation is to have a full connector configuration saved by the UI, so JSON/spec validation can verify that the connector's configuration (optionally in combination with injected OAuth params) is valid.

Handle injection for check/discover jobs too

sherifnada · 2021-09-08T18:27:15Z

airbyte-integrations/bases/base-normalization/entrypoint.sh

@@ -29,6 +29,8 @@ function configuredbt() {
    echo "Running: transform-config --config ${CONFIG_FILE} --integration-type ${INTEGRATION_TYPE} --out ${PROJECT_DIR}"
    # Generate a profiles.yml file for the selected destination/integration type
    transform-config --config "${CONFIG_FILE}" --integration-type "${INTEGRATION_TYPE}" --out "${PROJECT_DIR}"
+    # Remove config file as it might still contain sensitive credentials
+    rm "${CONFIG_FILE}"


SGTM - could you add a source code comment explaining why that file containing sensitive credentials is a bad thing and therefore it should be removed?

* Inject masked OAuth params for UI Validation * Add tests

github-actions bot added area/api Related to the api area/platform issues related to the platform area/documentation Improvements or additions to documentation labels Aug 31, 2021

ChristopheDuong commented Aug 31, 2021

View reviewed changes

airbyte-api/src/main/openapi/config.yaml Outdated Show resolved Hide resolved

Base automatically changed from chris/oauth-request-api to master August 31, 2021 22:21

Add a config for instance wide oauth parameters

e17bd6c

ChristopheDuong force-pushed the chris/oauth-params branch from 9db514c to e17bd6c Compare September 1, 2021 09:10

ChristopheDuong added 3 commits September 1, 2021 21:28

Remove recreate endpoints and add tests

a6128c5

Remove webbackend endpoints

c196c98

Scheduler injects oauth Parameters

6fdefce

ChristopheDuong marked this pull request as ready for review September 2, 2021 10:51

ChristopheDuong requested review from sherifnada, airbyte-jenny and jrhizor September 2, 2021 10:51

Refactor JsonNode merge

d4184d3

ChristopheDuong commented Sep 2, 2021

View reviewed changes

Remove comment

345ea46

sherifnada reviewed Sep 3, 2021

View reviewed changes

ChristopheDuong added 3 commits September 6, 2021 12:38

Merge remote-tracking branch 'origin/master' into oauth-params

087b149

Clean up recreate endpoints

44747c5

Inject oauth params in discover/check/reset too

c970466

github-actions bot added the normalization label Sep 6, 2021

ChristopheDuong commented Sep 6, 2021

View reviewed changes

sherifnada reviewed Sep 7, 2021

View reviewed changes

jrhizor reviewed Sep 7, 2021

View reviewed changes

ChristopheDuong added 3 commits September 8, 2021 11:35

make source-bigquery compile

e25cd47

Revert injecting in DefaultJobCreator back to DefaultSyncJobFactory

42d8425

Handle injection for check/discover jobs too

Merge remote-tracking branch 'origin/master' into oauth-params

2cd5617

github-actions bot added area/connectors Connector related issues and removed area/connectors Connector related issues labels Sep 8, 2021

Format code

340f79d

ChristopheDuong mentioned this pull request Sep 8, 2021

Inject masked OAuth params for UI Validation #5865

Merged

38 tasks

sherifnada approved these changes Sep 8, 2021

View reviewed changes

sherifnada and others added 3 commits September 9, 2021 00:15

conflict

ea7d992

Inject masked OAuth params for UI Validation (#5865)

cc7659a

* Inject masked OAuth params for UI Validation * Add tests

Add comment

46f3fee

ChristopheDuong merged commit edcd83d into master Sep 9, 2021

ChristopheDuong deleted the chris/oauth-params branch September 9, 2021 08:36

jrhizor mentioned this pull request Sep 16, 2021

Bump Airbyte version from 0.29.17-alpha to 0.29.18-alpha #6125

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a config for instance wide oauth parameters #5761

Add a config for instance wide oauth parameters #5761

ChristopheDuong commented Aug 31, 2021 •

edited

Loading

ChristopheDuong Sep 2, 2021 •

edited

Loading

sherifnada Sep 2, 2021

jrhizor commented Sep 2, 2021

sherifnada left a comment

sherifnada Sep 2, 2021

ChristopheDuong Sep 6, 2021

sherifnada Sep 7, 2021

sherifnada Sep 2, 2021

sherifnada Sep 2, 2021

ChristopheDuong Sep 6, 2021 •

edited

Loading

ChristopheDuong Sep 6, 2021

ChristopheDuong Sep 6, 2021

sherifnada Sep 7, 2021 •

edited

Loading

ChristopheDuong Sep 8, 2021 •

edited

Loading

sherifnada Sep 8, 2021

sherifnada left a comment •

edited

Loading

sherifnada Sep 7, 2021

sherifnada Sep 7, 2021 •

edited

Loading

sherifnada Sep 7, 2021

jrhizor Sep 7, 2021

sherifnada left a comment •

edited

Loading

jrhizor Sep 7, 2021

ChristopheDuong Sep 8, 2021 •

edited

Loading

jrhizor Sep 7, 2021

jrhizor Sep 7, 2021

jrhizor Sep 7, 2021

ChristopheDuong Sep 8, 2021 •

edited

Loading

sherifnada Sep 8, 2021

		final DestinationConnection destinationConnection = configRepository.getDestinationConnection(standardSync.getDestinationId());

		destinationConnection.withConfiguration(oAuthConfigSupplier.injectDestinationOAuthParameters(

Add a config for instance wide oauth parameters #5761

Add a config for instance wide oauth parameters #5761

Conversation

ChristopheDuong commented Aug 31, 2021 • edited Loading

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

ChristopheDuong Sep 2, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jrhizor commented Sep 2, 2021

sherifnada left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChristopheDuong Sep 6, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sherifnada Sep 7, 2021 • edited Loading

Choose a reason for hiding this comment

ChristopheDuong Sep 8, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sherifnada left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sherifnada Sep 7, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sherifnada left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChristopheDuong Sep 8, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChristopheDuong Sep 8, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChristopheDuong commented Aug 31, 2021 •

edited

Loading

ChristopheDuong Sep 2, 2021 •

edited

Loading

ChristopheDuong Sep 6, 2021 •

edited

Loading

sherifnada Sep 7, 2021 •

edited

Loading

ChristopheDuong Sep 8, 2021 •

edited

Loading

sherifnada left a comment •

edited

Loading

sherifnada Sep 7, 2021 •

edited

Loading

sherifnada left a comment •

edited

Loading

ChristopheDuong Sep 8, 2021 •

edited

Loading

ChristopheDuong Sep 8, 2021 •

edited

Loading