-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
source-mysql: do not use testcontainers
in integration tests
#30032
Conversation
Before Merging a Connector Pull RequestWow! What a great pull request you have here! 🎉 To merge this PR, ensure the following has been done/considered for each connector added or updated:
If the checklist is complete, but the CI check is failing,
|
1157bba
to
78d1257
Compare
d8e43b6
to
76ef3e6
Compare
.put(JdbcUtils.HOST_KEY, innerContainerAddress.left) | ||
.put(JdbcUtils.PORT_KEY, innerContainerAddress.right) | ||
.put(JdbcUtils.HOST_KEY, HostPortResolver.resolveHost(container)) | ||
.put(JdbcUtils.PORT_KEY, HostPortResolver.resolvePort(container)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this still work when running directly on macOS with gradle :integrationTestJava
?
257ac69
to
4843c27
Compare
aafe73c
to
36b6164
Compare
e4209fb
to
693dc87
Compare
testcontainers
in integration tests
/legacy-test connector=connectors/source-mysql
Build FailedTest summary info:
|
Closing as this is was fixed in other PRs |
Original problems:
Using
testcontainers
within airbyte-ci and locally has proven to be complicated:testcontainers
used an "old"mysql image which did not have an ARM64 variant. Updatingtestcontainers
solves the problem.testcontainers
containers andsource-mysql
containers are running on a sidecar docker host which is not the same host as where the test suite is executed. It means that client code (the tests) can reach themysql
db container using the sidecar dockerhost hostname, butsource-mysql
can't resolve this host name because it's not running inhost
network mode: it has to reach themysql
container using its IP. It introduces additional networking challenges in intregration tests, when the connector container has to communicate with the db connector. The difference with local execution of the integration test is that in local execution the docker host == localhost.testcontainers
spawns a lot of ephemeral database containers. It is resource intensive and makes running local / airbyte-ci hard.source-mysql
integration test suite has a lot of duplicated code, which makes it hard to understand the test flow and context.Suggested approach
testcontainers
for integration test: use a real remote DB provisionned on CloudSQL (GCP)Leave the the campground cleaner than you found it
/root/.gradle/
directory to a cache volumeOpen questions for java source developers
JdbcSourceAcceptanceTest
test suite as part of the unit test? Are you running this test suite locally when iterating on a connector? This test suite usestestcontainers
. It has no networking issue while running in airbyte-ci because these tests are not launching a connector as a container but use theMySqlSource
object that can directly connect . But as, I mentioned above, usingtestcontainers
adds a lot of overhead to test execution, can we use the remote test instance?Current blockers
**The java integration tests are hanging. When running
airbyte-ci connectors --name=source-mysql test
everything runs fine until the integration tests. Then the integration tests starts but the execution of./gradlew --no-daemon --scan --build-cache --debug :airbyte-integrations:connectors:source-mysql:integrationTest -x airbyteDocker
is not returning any progress while eating a ton of CPU. **TODO
source-mysql-strict-encrypt
gradle-cache
volume persisted by Dagger's magicache to benefit from it in the CI.