-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
airbyte-ci: fix integration test log buffering #30438
airbyte-ci: fix integration test log buffering #30438
Conversation
Before Merging a Connector Pull RequestWow! What a great pull request you have here! 🎉 To merge this PR, ensure the following has been done/considered for each connector added or updated:
If the checklist is complete, but the CI check is failing,
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Congrats for finding the root cause!
airbyte-ci/connectors/pipelines/pipelines/actions/environments.py
Outdated
Show resolved
Hide resolved
destination-snowflake test report (commit
|
Step | Result |
---|---|
Build connector tar | ✅ |
Build destination-snowflake docker image for platform linux/x86_64 | ✅ |
Java Connector Unit Tests | ✅ |
Java Connector Integration Tests | ✅ |
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml | ✅ |
Connector version semver check | ❌ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-snowflake test
Green for source-google-sheets as well: https://github.com/airbytehq/airbyte/actions/runs/6187637825/job/16797910130 |
destination-snowflake test report (commit
|
Step | Result |
---|---|
Build connector tar | ✅ |
Build destination-snowflake docker image for platform linux/x86_64 | ✅ |
Java Connector Unit Tests | ✅ |
Java Connector Integration Tests | ✅ |
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml | ✅ |
Connector version semver check | ❌ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-snowflake test
destination-snowflake test report (commit
|
Step | Result |
---|---|
Build connector tar | ✅ |
Build destination-snowflake docker image for platform linux/x86_64 | ✅ |
Java Connector Unit Tests | ✅ |
Java Connector Integration Tests | ✅ |
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml | ✅ |
Connector version semver check | ❌ |
QA checks | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-snowflake test
/approve-and-merge reason="not waiting for legacy CI" |
Thanks for the review! I've addressed all comments and ran some extra tests. |
This PR fixes the log buffering issue in airbyte-ci that has plagued our recent source-mysql integration tests debugging efforts. What would happen is that we'd never see the
./gradlew ...:integrationTest
output in a timely manner, unlike all other pipeline steps.As it turns out, the trigger for this buffering is that this container has secrets that were mounted to it. These secrets are the credentials downloaded from GSM so they are legit secrets. There's no way around this buffering, see commentary I added in the PR.
In any case, this PR adds a workaround when developing locally: we don't mount those files as secrets, just regular files. This is fine locally because those secrets are readily available on a development machine anyway. When airbyte-ci is not running locally, it's running in a github action or something and the buffering is something we can put up with in that case. In fact, so far, this hasn't been an issue at all in that context.
This PR includes another commit which has a trivial gradle fix. Its main purpose is to exercise this change on the snowflake connector. I'll do a couple of manual runs on python connectors just to make sure I didn't break anything there.