Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

airbyte-ci: fix integration test log buffering #30438

Conversation

postamar
Copy link
Contributor

@postamar postamar commented Sep 14, 2023

This PR fixes the log buffering issue in airbyte-ci that has plagued our recent source-mysql integration tests debugging efforts. What would happen is that we'd never see the ./gradlew ...:integrationTest output in a timely manner, unlike all other pipeline steps.

As it turns out, the trigger for this buffering is that this container has secrets that were mounted to it. These secrets are the credentials downloaded from GSM so they are legit secrets. There's no way around this buffering, see commentary I added in the PR.

In any case, this PR adds a workaround when developing locally: we don't mount those files as secrets, just regular files. This is fine locally because those secrets are readily available on a development machine anyway. When airbyte-ci is not running locally, it's running in a github action or something and the buffering is something we can put up with in that case. In fact, so far, this hasn't been an issue at all in that context.

This PR includes another commit which has a trivial gradle fix. Its main purpose is to exercise this change on the snowflake connector. I'll do a couple of manual runs on python connectors just to make sure I didn't break anything there.

@postamar postamar requested a review from a team as a code owner September 14, 2023 15:20
@postamar postamar requested review from alafanechere and a team and removed request for a team September 14, 2023 15:20
@github-actions
Copy link
Contributor

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

  • PR name follows PR naming conventions
  • Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
  • Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
  • You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
  • Secrets in the connector's spec are annotated with airbyte_secret
  • All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
  • Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
  • Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
  • If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

  1. Check for hidden checklists in your PR description

  2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

Copy link
Contributor

@alafanechere alafanechere left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Congrats for finding the root cause!

@github-actions
Copy link
Contributor

destination-snowflake test report (commit 2a2986df14) - ❌

⏲️ Total pipeline duration: 13mn15s

Step Result
Build connector tar
Build destination-snowflake docker image for platform linux/x86_64
Java Connector Unit Tests
Java Connector Integration Tests
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml
Connector version semver check
QA checks

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

@postamar
Copy link
Contributor Author

Green for source-google-sheets as well: https://github.com/airbytehq/airbyte/actions/runs/6187637825/job/16797910130
I also ran locally to make sure I didn't break things there.

@github-actions
Copy link
Contributor

destination-snowflake test report (commit ef36fa50f7) - ❌

⏲️ Total pipeline duration: 11mn04s

Step Result
Build connector tar
Build destination-snowflake docker image for platform linux/x86_64
Java Connector Unit Tests
Java Connector Integration Tests
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml
Connector version semver check
QA checks

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

@github-actions
Copy link
Contributor

destination-snowflake test report (commit 1825d6ca8a) - ❌

⏲️ Total pipeline duration: 03mn04s

Step Result
Build connector tar
Build destination-snowflake docker image for platform linux/x86_64
Java Connector Unit Tests
Java Connector Integration Tests
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml
Connector version semver check
QA checks

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

@postamar
Copy link
Contributor Author

/approve-and-merge reason="not waiting for legacy CI"

@postamar
Copy link
Contributor Author

Thanks for the review! I've addressed all comments and ran some extra tests.

@octavia-approvington
Copy link
Contributor

You did it!
finish line

@octavia-approvington octavia-approvington merged commit 532ef09 into master Sep 14, 2023
15 of 19 checks passed
@octavia-approvington octavia-approvington deleted the postamar/fix-airbyte-ci-integration-test-log-buffering branch September 14, 2023 16:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants