mySQL -> Snowflake Ingestion gets stuck in between mySQL to Snowflake transition

[tylerdelange]

Current Behavior

Usually, but not always, after the mySQL connection closes, the connection run just completely stalls in "Running" status. I have had a running connection now for several hours and it is not moving to the following step (either transferring to Debezium or Snowflake connector). It occurs to be happening primarily with larger tables. 50M + records.

I am do a mySQL (CDC) to Snowflake (using S3 staging) connection.

Also, I tried both resetting the connection and completely deleting the connection and re-creating it. The same problems occured.

Logs

Server Logs
ServerLogs.txt

mySQL Logs (--tail)
2021-06-24 17:51:26 INFO i.a.i.s.j.AbstractJdbcSource(lambda$read$4):193 - {} - Closing database connection pool.
2021-06-24 17:51:26 INFO i.a.i.s.j.AbstractJdbcSource(lambda$read$4):195 - {} - Closed database connection pool.
2021-06-24 17:51:26 INFO i.a.i.b.IntegrationRunner(run):118 - {} - Completed integration: io.airbyte.integrations.source.mysql.MySqlSource
2021-06-24 17:51:26 INFO i.a.i.s.m.MySqlSource(main):314 - {} - completed source: class io.airbyte.integrations.source.mysql.MySqlSource

Snowflake Logs
snowflakeLogs.txt

Steps to Reproduce

1. Create a mySQL source with CDC
2. Create Snowflake with S3 staging destination
3. Attempt ingestion of larger table (50 - 100M records)

Severity of the bug for you

Very High

Connector Version (if applicable)

airbyte 0.26.2-alpha
mySQL 0.3.8
snowflake 0.3.9

Additional context

Running the container on an extremely large "m5.16xlarge" instance with 1 TB of storage.

[danieldiamond]

might be related #5277

[danieldiamond]

additional information. sync just hangs at

2021-10-19 12:11:00 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-19 12:11:00 INFO i.a.i.s.r.AbstractDbSource(lambda$read$2):121 - {} - Closing database connection pool.
2021-10-19 12:11:00 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-19 12:11:00 INFO i.a.i.s.r.AbstractDbSource(lambda$read$2):123 - {} - Closed database connection pool.
2021-10-19 12:11:00 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-19 12:11:00 INFO i.a.i.b.IntegrationRunner(run):133 - {} - Completed integration: io.airbyte.integrations.base.ssh.SshWrappedSource
2021-10-19 12:11:00 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2021-10-19 12:11:00 INFO i.a.i.s.m.MySqlSource(main):249 - {} - completed source: class io.airbyte.integrations.source.mysql.MySqlSource

250M rows across ~15 tables
airbyte EC2 0.30.20-alpha
connector versions in specs below

CONTAINER ID   NAME                 CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
b5c61934202f   recursing_margulis   100.25%   2.821GiB / 30.96GiB   9.11%     0B / 0B           240MB / 3.63GB    42
14dc4b2f7d4c   elated_wilbur        0.15%     1.667GiB / 30.96GiB   5.38%     0B / 0B           228MB / 0B        369
73c72f499fb1   airbyte-server       0.08%     1.12GiB / 30.96GiB    3.62%     57.1GB / 6.09GB   355MB / 0B        176
2e5b9bfef778   airbyte-scheduler    0.04%     694.5MiB / 30.96GiB   2.19%     198GB / 2.28GB    307MB / 0B        59
e301c04538d1   airbyte-db           0.00%     231.7MiB / 30.96GiB   0.73%     4.11GB / 256GB    53.4MB / 209kB    50
f1deb8ddd649   airbyte-webapp       0.00%     13.17MiB / 30.96GiB   0.04%     5.02GB / 5.04GB   31.9MB / 12.3kB   9
0119fd378f23   airbyte-worker       0.07%     2.248GiB / 30.96GiB   7.26%     352MB / 295MB     535MB / 0B        138
81778d3d754e   airbyte-temporal     0.99%     228.4MiB / 30.96GiB   0.72%     1.31GB / 2.01GB   404MB / 24.6kB    17

CONTAINER ID   IMAGE                                  COMMAND                  CREATED       STATUS                  PORTS                                                                                         NAMES
b5c61934202f   airbyte/source-mysql:0.4.8             "/airbyte/base.sh re…"   7 hours ago   Up 7 hours                                                                                                            recursing_margulis
14dc4b2f7d4c   airbyte/destination-snowflake:0.3.16   "/airbyte/base.sh wr…"   7 hours ago   Up 7 hours                                                                                                            elated_wilbur
73c72f499fb1   airbyte/server:0.30.20-alpha           "/bin/bash -c bin/${…"   4 days ago    Up 4 days               8000/tcp, 0.0.0.0:8001->8001/tcp, :::8001->8001/tcp                                           airbyte-server
2e5b9bfef778   airbyte/scheduler:0.30.20-alpha        "/bin/bash -c bin/${…"   4 days ago    Up 4 days                                                                                                             airbyte-scheduler
e301c04538d1   airbyte/db:0.30.20-alpha               "docker-entrypoint.s…"   4 days ago    Up 4 days               5432/tcp                                                                                      airbyte-db
f1deb8ddd649   airbyte/webapp:0.30.20-alpha           "/docker-entrypoint.…"   4 days ago    Up 4 days               0.0.0.0:8000->80/tcp, :::8000->80/tcp                                                         airbyte-webapp
0119fd378f23   airbyte/worker:0.30.20-alpha           "/bin/bash -c bin/${…"   4 days ago    Up 4 days                                                                                                             airbyte-worker
ba4b42736bcb   airbyte/init:0.30.20-alpha             "/bin/sh -c './scrip…"   4 days ago    Exited (0) 4 days ago                                                                                                 init
81778d3d754e   temporalio/auto-setup:1.7.0            "/entrypoint.sh /bin…"   2 weeks ago   Up 4 days               6933-6935/tcp, 6939/tcp, 7234-7235/tcp, 7239/tcp, 0.0.0.0:7233->7233/tcp, :::7233->7233/tcp   airbyte-temporal

[grishick]

@danieldiamond we have upgraded Debezium version in MySQL and made many improvements to MySQL Source connector. Do you still see this problem with the latest version of MySQL Source Connector?

[grishick]

Closing the issue as unreproducible