-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
gradle: remove airbyteDocker.outputs dependencies #30314
gradle: remove airbyteDocker.outputs dependencies #30314
Conversation
source-dynamodb test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ❌ |
Build connector tar | ❌ |
Validate airbyte-integrations/connectors/source-dynamodb/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-dynamodb test
destination-redpanda test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ✅ |
Build connector tar | ✅ |
Build destination-redpanda docker image for platform linux/x86_64 | ✅ |
Java Connector Integration Tests | ❌ |
Validate airbyte-integrations/connectors/destination-redpanda/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-redpanda test
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like this change for airbyte-ci
. I did not dig into the gradle change. Please consider my connector testing comment.
@@ -436,7 +436,7 @@ async def get_connector_dir(self, exclude=None, include=None) -> Directory: | |||
Directory: The connector under test source code directory. | |||
""" | |||
vanilla_connector_dir = self.get_repo_dir(str(self.connector.code_directory), exclude=exclude, include=include) | |||
return await hacks.patch_connector_dir(self, vanilla_connector_dir) | |||
return await vanilla_connector_dir.with_timestamps(1) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think so: its a trick to avoid cache bursting when a file is rewritten without modification of its content.
from dagger import Client, Container, Directory | ||
from pipelines.contexts import ConnectorContext | ||
|
||
|
||
LINES_TO_REMOVE_FROM_GRADLE_FILE = [ | ||
# Do not build normalization with Gradle - we build normalization with Dagger in the BuildOrPullNormalization step. | ||
"project(':airbyte-integrations:bases:base-normalization').airbyteDocker.output", | ||
] | ||
|
||
|
||
async def _patch_gradle_file(context: ConnectorContext, connector_dir: Directory) -> Directory: | ||
"""Patch the build.gradle file of the connector under test by removing the lines declared in LINES_TO_REMOVE_FROM_GRADLE_FILE. | ||
|
||
Underlying issue: | ||
Java connectors build.gradle declare a dependency to the normalization module. | ||
It means every time we test a java connector the normalization is built. | ||
This is time consuming and not required as normalization is now baked in containers. | ||
Normalization is going away soon so hopefully this hack will be removed soon. | ||
|
||
Args: | ||
context (ConnectorContext): The initialized connector context. | ||
connector_dir (Directory): The directory containing the build.gradle file to patch. | ||
Returns: | ||
Directory: The directory containing the patched gradle file. | ||
""" | ||
if context.connector.language is not ConnectorLanguage.JAVA: | ||
context.logger.info(f"Connector language {context.connector.language} does not require a patched build.gradle file.") | ||
return connector_dir | ||
|
||
try: | ||
gradle_file_content = await connector_dir.file("build.gradle").contents() | ||
except DaggerError: | ||
context.logger.info("Could not find build.gradle file in the connector directory. Skipping patching.") | ||
return connector_dir | ||
|
||
context.logger.warn("Patching build.gradle file to remove normalization build.") | ||
|
||
patched_gradle_file = [] | ||
|
||
for line in gradle_file_content.splitlines(): | ||
if not any(line_to_remove in line for line_to_remove in LINES_TO_REMOVE_FROM_GRADLE_FILE): | ||
patched_gradle_file.append(line) | ||
return connector_dir.with_new_file("build.gradle", contents="\n".join(patched_gradle_file)) | ||
|
||
|
||
async def patch_connector_dir(context: ConnectorContext, connector_dir: Directory) -> Directory: | ||
"""Patch a connector directory: patch cat config, gradle file and dockerfile. | ||
|
||
Args: | ||
context (ConnectorContext): The initialized connector context. | ||
connector_dir (Directory): The directory containing the connector to patch. | ||
Returns: | ||
Directory: The directory containing the patched connector. | ||
""" | ||
patched_connector_dir = await _patch_gradle_file(context, connector_dir) | ||
return patched_connector_dir.with_timestamps(1) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I love seeing hacks being removed! In order to validate that it's not breaking anything I would love to see a test bhon the following "stable" java connectors:
source-postgres
, because it's a working source connectordestination-bigquery
because its a not using normalizationdestination-snowflake
@edgao @evantahler do you have a destination connector still using normalization which is passing test? I can't find any.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done! All green, modulo one small regression fix.
@@ -39,7 +38,6 @@ dependencies { | |||
|
|||
integrationTestJavaImplementation project(':airbyte-integrations:bases:standard-destination-test') | |||
integrationTestJavaImplementation project(':airbyte-integrations:connectors:destination-redshift') | |||
integrationTestJavaImplementation files(project(':airbyte-integrations:bases:base-normalization').airbyteDocker.outputs) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How do we declare the dependency of integration test on the build of the normalization image?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, nice catch. This is an oversight! I'll fix this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
destination-selectdb test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ❌ |
Build connector tar | ✅ |
Build destination-selectdb docker image for platform linux/x86_64 | ✅ |
Java Connector Integration Tests | ❌ |
Validate airbyte-integrations/connectors/destination-selectdb/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-selectdb test
destination-cassandra test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ✅ |
Build connector tar | ✅ |
Build destination-cassandra docker image for platform linux/x86_64 | ✅ |
Java Connector Integration Tests | ❌ |
Validate airbyte-integrations/connectors/destination-cassandra/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-cassandra test
destination-kafka test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ✅ |
Build connector tar | ✅ |
Build destination-kafka docker image for platform linux/x86_64 | ✅ |
Java Connector Integration Tests | ❌ |
Validate airbyte-integrations/connectors/destination-kafka/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-kafka test
destination-postgres test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ✅ |
Build connector tar | ✅ |
Build destination-postgres docker image for platform linux/x86_64 | ✅ |
Build airbyte/normalization:dev | ✅ |
Java Connector Integration Tests | ❌ |
Validate airbyte-integrations/connectors/destination-postgres/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-postgres test
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I completed a full read-through. Some is code I’m less familiar with, especially the legacy tasks and workflows. That said, I found no issues in my readthrough.
Thanks, @postamar, for putting this together. 👍
implementation project(':airbyte-integrations:bases:base-java') | ||
implementation files(project(':airbyte-integrations:bases:base-java').airbyteDocker.outputs) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice to see these being dropped. 👍
|
||
implementation files(project(':airbyte-integrations:bases:base-java').airbyteDocker.outputs) | ||
integrationTestJavaImplementation files(project(':airbyte-integrations:bases:base-normalization').airbyteDocker.outputs) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I mentioned this in our 1:1, but I’m excited to see these drop because they complicated the Java CDK build process. Happy to see them retiring.
source-cockroachdb test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ❌ |
Build connector tar | ✅ |
Build source-cockroachdb docker image for platform linux/x86_64 | ✅ |
Java Connector Integration Tests | ❌ |
Acceptance tests | ✅ |
Validate airbyte-integrations/connectors/source-cockroachdb/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-cockroachdb test
source-sftp test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ❌ |
Build connector tar | ❌ |
Validate airbyte-integrations/connectors/source-sftp/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-sftp test
source-cockroachdb-strict-encrypt test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ✅ |
Build connector tar | ✅ |
Build source-cockroachdb-strict-encrypt docker image for platform linux/x86_64 | ✅ |
Java Connector Integration Tests | ❌ |
Acceptance tests | ❌ |
Validate airbyte-integrations/connectors/source-cockroachdb-strict-encrypt/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-cockroachdb-strict-encrypt test
source-alloydb-strict-encrypt test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ✅ |
Build connector tar | ✅ |
Build source-alloydb-strict-encrypt docker image for platform linux/x86_64 | ✅ |
Java Connector Integration Tests | ✅ |
Acceptance tests | ❌ |
Validate airbyte-integrations/connectors/source-alloydb-strict-encrypt/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-alloydb-strict-encrypt test
destination-rockset test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ✅ |
Build connector tar | ✅ |
Build destination-rockset docker image for platform linux/x86_64 | ✅ |
Java Connector Integration Tests | ❌ |
Validate airbyte-integrations/connectors/destination-rockset/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-rockset test
destination-csv test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ✅ |
Build connector tar | ❌ |
Validate airbyte-integrations/connectors/destination-csv/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-csv test
destination-oracle test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ✅ |
Build connector tar | ✅ |
Build destination-oracle docker image for platform linux/x86_64 | ✅ |
Build airbyte/normalization-oracle:dev | ✅ |
Java Connector Integration Tests | ❌ |
Validate airbyte-integrations/connectors/destination-oracle/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-oracle test
destination-s3-glue test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ✅ |
Build connector tar | ✅ |
Build destination-s3-glue docker image for platform linux/x86_64 | ✅ |
Java Connector Integration Tests | ❌ |
Validate airbyte-integrations/connectors/destination-s3-glue/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-s3-glue test
destination-dev-null test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ✅ |
Build connector tar | ✅ |
Build destination-dev-null docker image for platform linux/x86_64 | ✅ |
Java Connector Integration Tests | ❌ |
Validate airbyte-integrations/connectors/destination-dev-null/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-dev-null test
destination-mongodb-strict-encrypt test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ✅ |
Build connector tar | ✅ |
Build destination-mongodb-strict-encrypt docker image for platform linux/x86_64 | ✅ |
Java Connector Integration Tests | ❌ |
Validate airbyte-integrations/connectors/destination-mongodb-strict-encrypt/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-mongodb-strict-encrypt test
destination-starburst-galaxy test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ❌ |
Build connector tar | ✅ |
Build destination-starburst-galaxy docker image for platform linux/x86_64 | ✅ |
Java Connector Integration Tests | ❌ |
Validate airbyte-integrations/connectors/destination-starburst-galaxy/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-starburst-galaxy test
source-clickhouse test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ✅ |
Build connector tar | ✅ |
Build source-clickhouse docker image for platform linux/x86_64 | ✅ |
Java Connector Integration Tests | ❌ |
Acceptance tests | ✅ |
Validate airbyte-integrations/connectors/source-clickhouse/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-clickhouse test
destination-yugabytedb test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ✅ |
Build connector tar | ✅ |
Build destination-yugabytedb docker image for platform linux/x86_64 | ✅ |
Java Connector Integration Tests | ❌ |
Validate airbyte-integrations/connectors/destination-yugabytedb/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-yugabytedb test
source-azure-blob-storage test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ✅ |
Build connector tar | ✅ |
Build source-azure-blob-storage docker image for platform linux/x86_64 | ✅ |
Java Connector Integration Tests | ❌ |
Acceptance tests | ❌ |
Validate airbyte-integrations/connectors/source-azure-blob-storage/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-azure-blob-storage test
destination-redshift test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ✅ |
Build connector tar | ✅ |
Build destination-redshift docker image for platform linux/x86_64 | ❌ |
Validate airbyte-integrations/connectors/destination-redshift/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-redshift test
destination-vertica test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ❌ |
Build connector tar | ❌ |
Validate airbyte-integrations/connectors/destination-vertica/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-vertica test
source-postgres-strict-encrypt test report (commit
|
Step | Result |
---|---|
Java Connector Unit Tests | ✅ |
Build connector tar | ✅ |
Build source-postgres-strict-encrypt docker image for platform linux/x86_64 | ✅ |
Java Connector Integration Tests | ✅ |
Acceptance tests | ✅ |
Validate airbyte-integrations/connectors/source-postgres-strict-encrypt/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=source-postgres-strict-encrypt test
This PR supersedes #30299 and is motivated by wanting to get rid of the gradle file patching hacks in airbyte-ci.