Read the information from both the definition file and the spec and add logs if have difference #19799

andriikorotkov · 2022-11-24T19:30:15Z

What

Added the ability to read the name and tag for the normalization container docker from the definition file, and compare this value with the value obtained from the NormalizationRunnerFactory. In case of difference, the message is added to the logs.

How

Describe the solution

🚨 User Impact 🚨

No impact

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

…DBT and normalizationImage fields. Added to the GenerateInputActivityImpl and TemporalClient classes code parts for read destination_definition.yaml and get suportDBT and normalizationImage fields. Added logging and comparing normalization images from NormalizationRunnerFactory and destination_definition.yaml

github-actions · 2022-11-24T19:31:30Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

✅ Sources (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

❌ Destinations (47)

Connector	Version	Changelog	Publish
`destination-aws-datalake`	`0.1.1`	✅	✅
`destination-azure-blob-storage`	`0.1.6`	⚠ (doc not found)	✅
`destination-bigquery`	`1.2.8`	✅	✅
`destination-bigquery-denormalized`	`1.2.8`	⚠ (doc not found)	✅
`destination-cassandra`	`0.1.4`	❌ (changelog missing)	✅
`destination-clickhouse`	`0.2.0`	✅	✅
`destination-clickhouse-strict-encrypt`	`0.2.0`	✅	⚠ (not in seed)
`destination-csv`	`0.2.10`	⚠ (doc not found)	✅
`destination-databricks`	`0.3.1`	✅	✅
`destination-dev-null`	`0.2.7`	⚠ (doc not found)	⚠ (not in seed)
`destination-doris`	`0.1.0`	❌ (changelog missing)	✅
`destination-dynamodb`	`0.1.7`	✅	✅
`destination-e2e-test`	`0.2.4`	✅	✅
`destination-elasticsearch`	`0.1.6`	✅	✅
`destination-elasticsearch-strict-encrypt`	`0.1.6`	✅	⚠ (not in seed)
`destination-gcs`	`0.2.12`	✅	✅
`destination-iceberg`	`0.1.0`	❌ (changelog missing)	✅
`destination-jdbc`	`0.3.14`	⚠ (doc not found)	⚠ (not in seed)
`destination-kafka`	`0.1.10`	❌ (changelog missing)	✅
`destination-keen`	`0.2.4`	❌ (changelog missing)	✅
`destination-kinesis`	`0.1.5`	❌ (changelog missing)	✅
`destination-local-json`	`0.2.11`	✅	✅
`destination-mariadb-columnstore`	`0.1.7`	✅	✅
`destination-mongodb`	`0.1.9`	✅	✅
`destination-mongodb-strict-encrypt`	`0.1.9`	✅	⚠ (not in seed)
`destination-mqtt`	`0.1.3`	✅	✅
`destination-mssql`	`0.1.22`	✅	✅
`destination-mssql-strict-encrypt`	`0.1.22`	✅	⚠ (not in seed)
`destination-mysql`	`0.1.20`	✅	✅
`destination-mysql-strict-encrypt`	❌ `0.1.21` (mismatch: `0.1.20`)	✅	⚠ (not in seed)
`destination-oracle`	`0.1.19`	✅	✅
`destination-oracle-strict-encrypt`	`0.1.19`	✅	⚠ (not in seed)
`destination-postgres`	`0.3.26`	✅	✅
`destination-postgres-strict-encrypt`	`0.3.26`	✅	⚠ (not in seed)
`destination-pubsub`	`0.1.6`	❌ (changelog missing)	✅
`destination-pulsar`	`0.1.3`	❌ (changelog missing)	✅
`destination-r2`	`0.1.0`	✅	✅
`destination-redis`	`0.1.4`	✅	✅
`destination-redpanda`	`0.1.0`	❌ (changelog missing)	✅
`destination-redshift`	`0.3.51`	✅	✅
`destination-rockset`	`0.1.4`	❌ (changelog missing)	✅
`destination-s3`	`0.3.17`	✅	✅
`destination-s3-glue`	`0.1.0`	❌ (changelog missing)	✅
`destination-scylla`	`0.1.3`	✅	✅
`destination-snowflake`	`0.4.40`	✅	✅
`destination-tidb`	`0.1.0`	✅	✅
`destination-yugabytedb`	`0.1.0`	✅	✅

See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. `source_definitions.yaml`), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the `/publish` command to publish the latest version.

pedroslopez

Thanks andrii! Didn't look too deeply at the logic, but my main concern at the moment is in using local definitions providers instead of reading the data directly from the database. More info in comments!

pedroslopez · 2022-11-24T19:43:45Z

airbyte-commons-temporal/src/main/java/io/airbyte/commons/temporal/TemporalClient.java

+      LocalDefinitionsProvider provider = new LocalDefinitionsProvider(LocalDefinitionsProvider.DEFAULT_SEED_DEFINITION_RESOURCE_CLASS);
+      List<StandardDestinationDefinition> destinationDefinitionList = provider.getDestinationDefinitions();


We shouldn't be reading from a definitions provider or direct yamls here- they're mainly used to seed/update the connector definitions in the database (actor_definitions table) and then any other operations within the platform should use the data in the db. For example, cloud doesn't use local definitions and instead loads remote definitions from a service.

I think by this point we should have the normalization tag / repo / anything else in that database table, specially since I see a migration adding those columns to the db... is this true?

Yes you're right. This is a very good point. Thank you for it. I changed this piece of code and now I get the data I need from the database. Also, I removed this logic from TemporalClient.class - as far as I understand, that part working on the synchronization process for which we may not retrieve this information.

...ommons-worker/src/main/java/io/airbyte/workers/normalization/NormalizationRunnerFactory.java

airbyte-worker-models/src/main/resources/workers_models/IntegrationLauncherConfig.yaml

...c/main/java/io/airbyte/workers/temporal/scheduling/activities/GenerateInputActivityImpl.java

pedroslopez

Mainly some questions!

Also, it would be great to see some tests added to make sure that we're retrieving the normalization image/tag as expected from the db

airbyte-commons-temporal/build.gradle

pedroslopez · 2022-11-28T13:51:36Z

...ommons-worker/src/main/java/io/airbyte/workers/normalization/DefaultNormalizationRunner.java

+  private static DestinationType getDestinationTypeByNormalizationImageNamePart(final String normalizationImageName) {
+    return EnumSet.allOf(DestinationType.class).stream()
+        .filter(destinationType -> normalizationImageName.contains(destinationType.name().toLowerCase()))
+        .findFirst()
+        .orElseGet(null);
+  }


Are we planning on / is there a way to remove this DestinationType enum? What is the destination type used for? I think leaving these in the platform means we still have some coupling and would require a platform change for adding normalization to a new connector, if I'm reading this correctly. Ideally we don't have anything connector-specific in the platform so they can evolve independently.

Currently we have an enum with specific destinations:

public enum DestinationType { BIGQUERY, MSSQL, MYSQL, ORACLE, POSTGRES, REDSHIFT, SNOWFLAKE, CLICKHOUSE, TIDB }

Everything you wrote about is true. I plan to remove the DestinationType enum in the next pull request, where the NormalizationRunnerFactory will also be removed. You must have been misled by the getDestinationTypeByNormalizationImageNamePart method. I will delete it.

...ommons-worker/src/main/java/io/airbyte/workers/normalization/NormalizationRunnerFactory.java

...ntainer-orchestrator/src/main/java/io/airbyte/container_orchestrator/DbtJobOrchestrator.java

pedroslopez · 2022-11-28T14:10:59Z

...c/main/java/io/airbyte/workers/temporal/scheduling/activities/GenerateInputActivityImpl.java

+      List<StandardDestinationDefinition> destinationDefinitionList = configRepository.listStandardDestinationDefinitions(true);
+      Optional<StandardDestinationDefinition> optionalDestinationDefinition = destinationDefinitionList.stream()
+          .filter(destinationDefinition -> config.getDestinationDockerImage()
+              .equalsIgnoreCase(destinationDefinition.getDockerRepository() + ":" + destinationDefinition.getDockerImageTag()))


note: there's a utility function DockerUtils.getTaggedImageName that deals with combining the repo and tag. Not the most complicated thing to do here but might as well use it if it's available.

I have not yet come across the use of this extra class. Thanks for sharing it)

...c/main/java/io/airbyte/workers/temporal/scheduling/activities/GenerateInputActivityImpl.java

…18232

pedroslopez

We should standardize to supportsDbt (with an s) but no need for another review from me for this change.

Please ensure that the tests in CI are passing before merging, seems like the branch is currently red.

Note: after this pr is merged we will still be using the platform-provided normalization image/versions but will start to read the definitions and log if there's a mismatch. A follow-up PR will actually remove the paltform-provided normalization images/versions and just use the ones from the definitions so I'm not expecting this to introduce a user-impacting change just yet.

pedroslopez · 2022-11-29T15:17:53Z

...ommons-worker/src/main/java/io/airbyte/workers/normalization/NormalizationRunnerFactory.java

+      log.error(
+          "The normalization image name or tag in the definition file is different from the normalization image or tag in the NormalizationRunnerFactory!");
+      log.error(
+          "the definition file value - {}, the NormalizationRunnerFactory value - {}", normalizationImage, factoryNormalizationImage);
+    }


is there any action to take if we get these logs?

currently - no, just to see that different versions are used. In general, this is necessary for testing and making sure that we are getting the correct data from the actor_definition table

pedroslopez · 2022-11-29T15:23:40Z

airbyte-api/src/main/openapi/config.yaml

+        supportDbt:
+          type: boolean
+          default: false


Suggested change

supportDbt:

type: boolean

default: false

supportsDbt:

type: boolean

default: false

nit to keep consistent

pedroslopez · 2022-11-29T15:23:52Z

airbyte-api/src/main/openapi/config.yaml

+        supportDbt:
+          type: boolean
+          default: false


Suggested change

supportDbt:

type: boolean

default: false

supportsDbt:

type: boolean

default: false

pedroslopez · 2022-11-29T15:24:15Z

airbyte-worker-models/src/main/resources/workers_models/IntegrationLauncherConfig.yaml

+  supportDbt:
+    type: boolean
+    default: false


Suggested change

supportDbt:

type: boolean

default: false

supportsDbt:

type: boolean

default: false

pedroslopez · 2022-11-29T15:26:21Z

airbyte-workers/src/main/java/io/airbyte/workers/temporal/sync/NormalizationActivityImpl.java

+    log.error("destination DBT -> {}", destinationLauncherConfig.getSupportDbt());
+    log.error("destination normalization -> {}", destinationLauncherConfig.getNormalizationDockerImage());


are these error logs intentional?

yes, I added them on purpose, but I'd probably rather remove them. They are needed more for development than for the end user

pedroslopez · 2022-11-29T15:29:11Z

airbyte-api/src/main/openapi/config.yaml

+        normalizationRepository:
+          type: string
+        normalizationTag:
+          type: string
+        supportDbt:
+          type: boolean
+          default: false


I think this api is called when adding custom destinations in OSS... do we need to add these to the form in the frontend?

What happens if a destination doesn't have these fields set? Are we properly handling this as a destination that doesn't run normalization?

…18232

…ec and add logs if have difference (#19799)" This reverts commit f11507b.

andriikorotkov requested review from evantahler, pedroslopez, alafanechere and DoNotPanicUA November 24, 2022 19:30

andriikorotkov linked an issue Nov 24, 2022 that may be closed by this pull request

Read the information from both the definition file and the spec and add logs if have difference. #18235

Closed

octavia-squidington-iv added area/platform issues related to the platform area/worker Related to worker labels Nov 24, 2022

andriikorotkov temporarily deployed to more-secrets November 24, 2022 19:31 Inactive

pedroslopez requested changes Nov 24, 2022

View reviewed changes

updated minor remarks

7caa00c

andriikorotkov temporarily deployed to more-secrets November 25, 2022 14:34 Inactive

updated minor remarks

8c46fd3

andriikorotkov temporarily deployed to more-secrets November 25, 2022 16:01 Inactive

andriikorotkov requested a review from pedroslopez November 25, 2022 16:18

pedroslopez reviewed Nov 28, 2022

View reviewed changes

fixed minor remarks

8c7f464

andriikorotkov temporarily deployed to more-secrets November 28, 2022 15:41 Inactive

evantahler mentioned this pull request Nov 29, 2022

Remove feature flags from ConnectorSpecification #19855

Closed

added normalization data to the tests

1d90e5d

octavia-squidington-iv added area/api Related to the api area/server labels Nov 29, 2022

andriikorotkov temporarily deployed to more-secrets November 29, 2022 11:42 Inactive

andriikorotkov added 2 commits November 29, 2022 14:42

fixed minor remarks

7f97179

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

980459c

…18232

andriikorotkov temporarily deployed to more-secrets November 29, 2022 12:44 Inactive

andriikorotkov requested a review from pedroslopez November 29, 2022 13:24

andriikorotkov marked this pull request as ready for review November 29, 2022 13:44

andriikorotkov temporarily deployed to more-secrets November 29, 2022 13:45 Inactive

pedroslopez approved these changes Nov 29, 2022

View reviewed changes

andriikorotkov linked an issue Nov 29, 2022 that may be closed by this pull request

Get normalization image from the connector definition file #18232

Closed

fixed remarks

e04c848

octavia-squidington-iv removed the area/api Related to the api label Nov 29, 2022

andriikorotkov temporarily deployed to more-secrets November 29, 2022 20:12 Inactive

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

a035df6

…18232

andriikorotkov temporarily deployed to more-secrets November 29, 2022 20:45 Inactive

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

cd65ebb

…18232

andriikorotkov temporarily deployed to more-secrets November 29, 2022 21:12 Inactive

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

7f72805

…18232

andriikorotkov temporarily deployed to more-secrets November 30, 2022 07:45 Inactive

andriikorotkov temporarily deployed to more-secrets November 30, 2022 07:46 Inactive

andriikorotkov temporarily deployed to more-secrets November 30, 2022 09:30 Inactive

andriikorotkov merged commit f11507b into master Nov 30, 2022

andriikorotkov deleted the akorotkov/18232 branch November 30, 2022 10:16

octavia-squidington-iii mentioned this pull request Nov 30, 2022

Bump Airbyte version from 0.40.22 to 0.40.23 #19941

Merged

malikdiarra added a commit that referenced this pull request Dec 5, 2022

Revert "Read the information from both the definition file and the sp…

3f3faea

…ec and add logs if have difference (#19799)" This reverts commit f11507b.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Read the information from both the definition file and the spec and add logs if have difference #19799

Read the information from both the definition file and the spec and add logs if have difference #19799

andriikorotkov commented Nov 24, 2022

github-actions bot commented Nov 24, 2022 •

edited

Loading

✅ Sources (0)

❌ Destinations (47)

✅ Other Modules (0)

Actionable Items

pedroslopez left a comment

pedroslopez Nov 24, 2022

andriikorotkov Nov 25, 2022

pedroslopez left a comment

pedroslopez Nov 28, 2022

pedroslopez Nov 28, 2022

andriikorotkov Nov 28, 2022

pedroslopez Nov 28, 2022

andriikorotkov Nov 28, 2022

pedroslopez left a comment

pedroslopez Nov 29, 2022

andriikorotkov Nov 29, 2022

pedroslopez Nov 29, 2022

pedroslopez Nov 29, 2022

pedroslopez Nov 29, 2022

pedroslopez Nov 29, 2022

andriikorotkov Nov 29, 2022

pedroslopez Nov 29, 2022

		LocalDefinitionsProvider provider = new LocalDefinitionsProvider(LocalDefinitionsProvider.DEFAULT_SEED_DEFINITION_RESOURCE_CLASS);
		List<StandardDestinationDefinition> destinationDefinitionList = provider.getDestinationDefinitions();

		log.error("destination DBT -> {}", destinationLauncherConfig.getSupportDbt());
		log.error("destination normalization -> {}", destinationLauncherConfig.getNormalizationDockerImage());

Read the information from both the definition file and the spec and add logs if have difference #19799

Read the information from both the definition file and the spec and add logs if have difference #19799

Conversation

andriikorotkov commented Nov 24, 2022

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

github-actions bot commented Nov 24, 2022 • edited Loading

Affected Connector Report

✅ Sources (0)

❌ Destinations (47)

✅ Other Modules (0)

Actionable Items

pedroslopez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pedroslopez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pedroslopez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Nov 24, 2022 •

edited

Loading