🐛 fixed OOM error when splitting a stream into several files #7074

andriikorotkov · 2021-10-15T13:10:54Z

What

AWS S3 Staging COPY is writing records from different table in the same raw table

How

It is optimal to write every 10,000,000 records to a new file. This will make it easier to work with files and speed up the recording of large amounts of data. In addition, for a large number of records, we will not get a drop in the copy request to QUERY_TIMEOUT when the records from the file are copied to the staging table.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

andriikorotkov · 2021-10-15T13:14:00Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1346052847
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1346052847
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     120      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 370    218    41%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         140     29    79%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1062    403    62%

andriikorotkov · 2021-10-15T13:14:15Z

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1346053543
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1346053543
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     120      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 370    218    41%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         140     29    79%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1062    403    62%

...tion-jdbc/src/main/java/io/airbyte/integrations/destination/jdbc/copy/s3/S3StreamCopier.java

tuliren · 2021-10-15T17:43:49Z

@andriikorotkov, the previous PR #6949 is titled as a fix to the bug that writes multiple streams to the same table. However, does it actually fix this bug? There was no unit test in that PR to show that the bug had been fixed. Could you add unit tests in this PR?

andriikorotkov · 2021-10-15T17:56:48Z

@tuliren, you are right, this is not entirely true. I saw that contributors in the troubleshooting group had an OOM error at this point. I tested this again with a lot of data (as you can see it in the screenshots) and fixed it in no time. If you have time, I would like to know your opinion on this fix and get any remarks from you.

andriikorotkov · 2021-10-15T18:48:23Z

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1347114192
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1347114192
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     120      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 370    218    41%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         140     29    79%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1062    403    62%

andriikorotkov · 2021-10-15T19:22:03Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1347213500
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1347213500
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     120      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 370    218    41%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         140     29    79%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1062    403    62%

...tion-jdbc/src/main/java/io/airbyte/integrations/destination/jdbc/copy/s3/S3StreamCopier.java

andriikorotkov · 2021-10-15T21:35:21Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1347573288

...tion-jdbc/src/main/java/io/airbyte/integrations/destination/jdbc/copy/s3/S3StreamCopier.java

tuliren · 2021-10-16T08:29:09Z

Thanks for adding the testSyncWithBillionRecords test case. However, it looks like it takes too long to run. I think it's fine to not include a load test here. We will solve the stress test somewhere else.

andriikorotkov · 2021-10-16T09:51:21Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1348878751
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1348878751
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     120      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 370    218    41%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         140     29    79%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1062    403    62%

andriikorotkov · 2021-10-16T09:51:32Z

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1348878868
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1348878868
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     120      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 370    218    41%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         140     29    79%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1062    403    62%

andriikorotkov · 2021-10-16T09:53:36Z

@tuliren I marked a test with a lot of records as disabled.

tuliren · 2021-10-18T09:30:28Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1353995297
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1353995297
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     120      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 370    218    41%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         140     29    79%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1062    403    62%

tuliren · 2021-10-18T09:30:35Z

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1353995661
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1353995661
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    12      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     120      6    95%
	 normalization/transform_catalog/reserved_keywords.py                 11      0   100%
	 normalization/transform_catalog/stream_processor.py                 370    218    41%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         140     29    79%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1062    403    62%

tuliren

I separated the filename generation logic to its own class, StagingFilenameGenerator, so that the logics can be unit tested. It turns out that there was an off-by-one bug in the original code, which has been fixed now.

andriikorotkov · 2021-10-18T10:27:59Z

/publish connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1354194548
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1354194548

andriikorotkov · 2021-10-18T11:06:05Z

/publish connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1354324703
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1354324703

…hq#7074) * UPDATED DESTINATION JDBC * fixed remarks * fixed remarks * added new test for testing largeNumberRecords * updated test * fixed remarks * Separate staging filename generator and add tests Co-authored-by: Liren Tu <tuliren.git@outlook.com>

UPDATED DESTINATION JDBC

553561e

andriikorotkov requested a review from alexandr-shegeda October 15, 2021 13:10

andriikorotkov temporarily deployed to more-secrets October 15, 2021 13:10 Inactive

github-actions bot added the area/connectors Connector related issues label Oct 15, 2021

andriikorotkov linked an issue Oct 15, 2021 that may be closed by this pull request

AWS S3 Staging COPY is writing records from different table in the same raw table #5664

Closed

jrhizor temporarily deployed to more-secrets October 15, 2021 13:16 Inactive

alexandr-shegeda reviewed Oct 15, 2021

View reviewed changes

...tion-jdbc/src/main/java/io/airbyte/integrations/destination/jdbc/copy/s3/S3StreamCopier.java Outdated Show resolved Hide resolved

andriikorotkov marked this pull request as ready for review October 15, 2021 17:49

andriikorotkov requested a review from tuliren October 15, 2021 17:49

andriikorotkov changed the title ~~bug AWS S3 Staging COPY is writing records from different table in the same raw table~~ 🐛 fixed OOM error when splitting a stream into several files Oct 15, 2021

fixed remarks

963d5a5

andriikorotkov temporarily deployed to more-secrets October 15, 2021 18:23 Inactive

andriikorotkov requested a review from alexandr-shegeda October 15, 2021 18:27

fixed remarks

5f24f18

andriikorotkov temporarily deployed to more-secrets October 15, 2021 18:32 Inactive

alexandr-shegeda approved these changes Oct 15, 2021

View reviewed changes

jrhizor temporarily deployed to more-secrets October 15, 2021 18:50 Inactive

jrhizor temporarily deployed to more-secrets October 15, 2021 19:24 Inactive

tuliren suggested changes Oct 15, 2021

View reviewed changes

...tion-jdbc/src/main/java/io/airbyte/integrations/destination/jdbc/copy/s3/S3StreamCopier.java Show resolved Hide resolved

added new test for testing largeNumberRecords

66595cd

andriikorotkov temporarily deployed to more-secrets October 15, 2021 21:36 Inactive

jrhizor temporarily deployed to more-secrets October 15, 2021 21:37 Inactive

andriikorotkov temporarily deployed to more-secrets October 16, 2021 08:15 Inactive

tuliren reviewed Oct 16, 2021

View reviewed changes

fixed remarks

ea4b1e0

andriikorotkov temporarily deployed to more-secrets October 16, 2021 09:52 Inactive

jrhizor temporarily deployed to more-secrets October 16, 2021 09:53 Inactive

andriikorotkov requested a review from tuliren October 16, 2021 10:01

Separate staging filename generator and add tests

9002c25

tuliren temporarily deployed to more-secrets October 18, 2021 09:30 Inactive

tuliren approved these changes Oct 18, 2021

View reviewed changes

jrhizor temporarily deployed to more-secrets October 18, 2021 09:32 Inactive

jrhizor temporarily deployed to more-secrets October 18, 2021 10:30 Inactive

jrhizor temporarily deployed to more-secrets October 18, 2021 11:08 Inactive

andriikorotkov merged commit ad2b6db into master Oct 18, 2021

andriikorotkov deleted the akorotkov|copy-s3-fix branch October 18, 2021 12:00

andriikorotkov linked an issue Oct 18, 2021 that may be closed by this pull request

Out of memory error when using S3 as staging storage for Snowflake #5277

Closed

jrhizor mentioned this pull request Oct 19, 2021

Bump Airbyte version from 0.30.20-alpha to 0.30.21-alpha #7145

Merged

sashaNeshcheret mentioned this pull request Oct 19, 2021

100% CPU utilization on CDC replication from Postgres CDC #6003

Closed

karinakuz added connectors/destinations-warehouse connectors/destination/snowflake labels Jan 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 fixed OOM error when splitting a stream into several files #7074

🐛 fixed OOM error when splitting a stream into several files #7074

andriikorotkov commented Oct 15, 2021

andriikorotkov commented Oct 15, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 15, 2021 •

edited by github-actions bot

Loading

tuliren commented Oct 15, 2021

andriikorotkov commented Oct 15, 2021

andriikorotkov commented Oct 15, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 15, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 15, 2021 •

edited by github-actions bot

Loading

tuliren commented Oct 16, 2021

andriikorotkov commented Oct 16, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 16, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 16, 2021

tuliren commented Oct 18, 2021 •

edited by github-actions bot

Loading

tuliren commented Oct 18, 2021 •

edited by github-actions bot

Loading

tuliren left a comment

andriikorotkov commented Oct 18, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 18, 2021 •

edited by github-actions bot

Loading

🐛 fixed OOM error when splitting a stream into several files #7074

🐛 fixed OOM error when splitting a stream into several files #7074

Conversation

andriikorotkov commented Oct 15, 2021

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

andriikorotkov commented Oct 15, 2021 • edited by github-actions bot Loading

andriikorotkov commented Oct 15, 2021 • edited by github-actions bot Loading

tuliren commented Oct 15, 2021

andriikorotkov commented Oct 15, 2021

andriikorotkov commented Oct 15, 2021 • edited by github-actions bot Loading

andriikorotkov commented Oct 15, 2021 • edited by github-actions bot Loading

andriikorotkov commented Oct 15, 2021 • edited by github-actions bot Loading

tuliren commented Oct 16, 2021

andriikorotkov commented Oct 16, 2021 • edited by github-actions bot Loading

andriikorotkov commented Oct 16, 2021 • edited by github-actions bot Loading

andriikorotkov commented Oct 16, 2021

tuliren commented Oct 18, 2021 • edited by github-actions bot Loading

tuliren commented Oct 18, 2021 • edited by github-actions bot Loading

tuliren left a comment

Choose a reason for hiding this comment

andriikorotkov commented Oct 18, 2021 • edited by github-actions bot Loading

andriikorotkov commented Oct 18, 2021 • edited by github-actions bot Loading

andriikorotkov commented Oct 15, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 15, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 15, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 15, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 15, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 16, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 16, 2021 •

edited by github-actions bot

Loading

tuliren commented Oct 18, 2021 •

edited by github-actions bot

Loading

tuliren commented Oct 18, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 18, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Oct 18, 2021 •

edited by github-actions bot

Loading