🎉 New Destination: R2 Cloudflare #15296

videnkz · 2022-08-04T12:46:41Z

What

Closes New Destination: R2 Cloudflare #13790
The connector is published in Publish R2 destination #17122.

🚨 User Impact 🚨

None expected.

Tests

Pre-merge checks

See Publish R2 destination #17122.

CLAassistant · 2022-08-04T12:46:48Z

All committers have signed the CLA.

marcosmarxm · 2022-08-04T14:20:07Z

@kananindzya let us know when this contribution is ready to review.

added base-java-s3 module updated common s3 lib local changed dependency of s3 in all realted modules minor style fixes added common module for s3 integration tests

…te into feature-r2-destination

…ation

…te into feature-r2-destination

videnkz · 2022-09-07T11:12:04Z

Should I delete PARQUET format from R2 specification and documentation?

That sounds good.

@kananindzya, I plan to merge this PR this week. Currently there are some merge conflicts. Would you mind granting me the write access to your fork? In that way, I can update this PR directly, and resolve some of the conflicts myself.

Hi @tuliren

I gave access to you.
Deleted parquet format from specification.
Updated branch
Added configuration for upload threads count. I've found limitations in R2 docs(https://developers.cloudflare.com/r2/platform/s3-compatibility/api/#implemented-object-level-operations).
On my environment I catch exception below:

022-09-03 02:06:55 �[43mdestination�[0m > 2022-09-03 02:06:55 �[1;31mERROR�[m a.m.s.StreamTransferManager(abort):432 - Aborting [Manager uploading to path/to/my/bucket/xxx/2022_09_03_1662168027012_.csv with id AMAM7OnPp...esYGAxA==] due to error: com.amazonaws.services.s3.model.AmazonS3Exception: There was a problem with the multipart upload. (Service: Amazon S3; Status Code: 400; Error Code: InvalidPart; Request ID: null; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null
2022-09-03 02:06:55 �[43mdestination�[0m > 2022-09-03 02:06:55 �[1;31mERROR�[m i.a.i.d.s.S3StorageOperations(uploadRecordsToBucket):128 - Failed to upload records into storage path/to/my/bucket/xxx/2022_09_03_1662168027012_
2022-09-03 02:06:55 �[43mdestination�[0m > java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.amazonaws.AbortedException: 
2022-09-03 02:06:55 �[43mdestination�[0m > 	at alex.mojaki.s3upload.ExecutorServiceResultsHandler$1.next(ExecutorServiceResultsHandler.java:72) ~[s3-stream-upload-2.2.2.jar:?]

videnkz · 2022-09-08T05:15:01Z

Should I delete PARQUET format from R2 specification and documentation?

That sounds good.
@kananindzya, I plan to merge this PR this week. Currently there are some merge conflicts. Would you mind granting me the write access to your fork? In that way, I can update this PR directly, and resolve some of the conflicts myself.

Hi @tuliren

I gave access to you.

Deleted parquet format from specification.

Updated branch

Added configuration for upload threads count. I've found limitations in R2 docs(https://developers.cloudflare.com/r2/platform/s3-compatibility/api/#implemented-object-level-operations).
On my environment I catch exception below:
022-09-03 02:06:55 �[43mdestination�[0m > 2022-09-03 02:06:55 �[1;31mERROR�[m a.m.s.StreamTransferManager(abort):432 - Aborting [Manager uploading to path/to/my/bucket/xxx/2022_09_03_1662168027012_.csv with id AMAM7OnPp...esYGAxA==] due to error: com.amazonaws.services.s3.model.AmazonS3Exception: There was a problem with the multipart upload. (Service: Amazon S3; Status Code: 400; Error Code: InvalidPart; Request ID: null; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null
2022-09-03 02:06:55 �[43mdestination�[0m > 2022-09-03 02:06:55 �[1;31mERROR�[m i.a.i.d.s.S3StorageOperations(uploadRecordsToBucket):128 - Failed to upload records into storage path/to/my/bucket/xxx/2022_09_03_1662168027012_
2022-09-03 02:06:55 �[43mdestination�[0m > java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.amazonaws.AbortedException: 
2022-09-03 02:06:55 �[43mdestination�[0m > 	at alex.mojaki.s3upload.ExecutorServiceResultsHandler$1.next(ExecutorServiceResultsHandler.java:72) ~[s3-stream-upload-2.2.2.jar:?]

Hi @tuliren ,
Today I tested multipart upload with concurrent thread count = 3.
Uploading finished success for file with 30mb size(part size - 5mb)

tuliren · 2022-09-26T00:15:04Z

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/3124227743
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/3124227743
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         191     49    74%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 589    394    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1437    624    57%

Build Passed

Test summary info:

All Passed

tuliren · 2022-09-26T00:15:15Z

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/3124228368
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/3124228368
No Python unittests run

Build Passed

Test summary info:

All Passed

tuliren · 2022-09-26T00:34:16Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/3124294249
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/3124294249
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         191     49    74%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 589    394    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1437    624    57%

Build Passed

Test summary info:

All Passed

tuliren · 2022-09-26T00:42:14Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/3124317178
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/3124317178
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         191     49    74%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 589    394    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1437    624    57%

Build Passed

Test summary info:

All Passed

tuliren · 2022-09-26T00:45:21Z

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/3124326322
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/3124326322
No Python unittests run

Build Passed

Test summary info:

All Passed

tuliren · 2022-09-26T00:45:27Z

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/3124326503
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/3124326503
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         191     49    74%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 589    394    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1437    624    57%

Build Passed

Test summary info:

All Passed

tuliren · 2022-09-26T00:49:25Z

/test connector=connectors/destination-databricks

🕑 connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/3124340116
✅ connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/3124340116
No Python unittests run

Build Passed

Test summary info:

All Passed

tuliren · 2022-09-26T02:20:28Z

All integration tests have passed. I will merge this PR, and publish the connector in a separate one.

tuliren · 2022-09-26T02:49:37Z

@kananindzya, the R2 connector has been published to docker:
https://hub.docker.com/r/airbyte/destination-r2/tags

Thanks a lot for your contribution.

Don't forget to remove me from your repo.

* added destination-r2 spec added base-java-s3 module updated common s3 lib local changed dependency of s3 in all realted modules minor style fixes added common module for s3 integration tests * fix configuration for r2 integration tests * fix configuration for r2 integration tests * added docs for destination-r2 * minor import fixes * mark test as disabled * fixed imports in destination-snowflake * restored styling task * added tests for S3DEstinationConfig * added upload threads count configuration for r2(due to limitations) * deleted parquet format type * Fix import in gcs * Fix import in redshift * Fix import in snowflake * Fix one more import in gcs * Fix one more import in redshift * Fix import in databricks Co-authored-by: Liren Tu <tuliren.git@outlook.com>

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Aug 4, 2022

octavia-squidington-iii added community connectors/destination/r2 labels Aug 4, 2022

marcosmarxm changed the title ~~destination-r2~~ 🎉 New Destination: R2 Aug 4, 2022

marcosmarxm changed the title ~~🎉 New Destination: R2~~ 🎉 New Destination: R2 Cloudflare Aug 4, 2022

octavia-squidington-iii added connectors/destination/s3 connectors/destination/jdbc labels Aug 8, 2022

added destination-r2 spec

bc59ea5

added base-java-s3 module updated common s3 lib local changed dependency of s3 in all realted modules minor style fixes added common module for s3 integration tests

videnkz force-pushed the feature-r2-destination branch from ffd9ca3 to bc59ea5 Compare August 11, 2022 09:08

octavia-squidington-iii added connectors/destination/bigquery connectors/destination/bigquery-denormalized connectors/destination/snowflake connectors/destination/redshift connectors/destination/databricks connectors/destination/gcs labels Aug 11, 2022

videnkz and others added 10 commits August 11, 2022 15:13

merge conflict resolved

637d9dd

fix configuration for r2 integration tests

f837cb4

fix configuration for r2 integration tests

164cd50

Merge branch 'feature-r2-destination' of github.com:kananindzya/airby…

5aa6cac

…te into feature-r2-destination

Merge remote-tracking branch 'upstream/master' into feature-r2-destin…

95d2e0d

…ation

Merge branch 'master' into feature-r2-destination

9c357f8

added docs for destination-r2

d9aa5d4

Merge branch 'master' into feature-r2-destination

6254afb

minor import fixes

ebc4adb

mark test as disabled

b3a81be

videnkz marked this pull request as ready for review August 12, 2022 10:31

videnkz requested a review from a team as a code owner August 12, 2022 10:31

videnkz and others added 2 commits September 7, 2022 17:07

Merge branch 'master' into feature-r2-destination

1a70acf

Merge branch 'feature-r2-destination' of github.com:kananindzya/airby…

20ee5f0

…te into feature-r2-destination

Merge branch 'master' into feature-r2-destination

b3043ff

tuliren approved these changes Sep 26, 2022

View reviewed changes

tuliren added 3 commits September 25, 2022 17:37

Fix import in gcs

657818a

Fix import in redshift

dbb5659

Fix import in snowflake

7cd007f

tuliren added 2 commits September 25, 2022 17:43

Fix one more import in gcs

ff611c9

Fix one more import in redshift

ef315bb

Fix import in databricks

761d53a

tuliren merged commit 3494405 into airbytehq:master Sep 26, 2022

tuliren deleted the feature-r2-destination branch September 26, 2022 02:20

tuliren mentioned this pull request Sep 26, 2022

Publish R2 destination #17122

Merged

18 tasks

This was referenced Sep 27, 2022

Bump Airbyte version from 0.40.9 to 0.40.10 #17285

Closed

Bump Airbyte version from 0.40.9 to 0.40.10 #17359

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Destination: R2 Cloudflare #15296

🎉 New Destination: R2 Cloudflare #15296

videnkz commented Aug 4, 2022 •

edited by tuliren

Loading

CLAassistant commented Aug 4, 2022 •

edited

Loading

marcosmarxm commented Aug 4, 2022

videnkz commented Sep 7, 2022 •

edited

Loading

videnkz commented Sep 8, 2022

tuliren commented Sep 26, 2022 •

edited by github-actions bot

Loading

tuliren commented Sep 26, 2022 •

edited by github-actions bot

Loading

tuliren commented Sep 26, 2022 •

edited by github-actions bot

Loading

tuliren commented Sep 26, 2022 •

edited by github-actions bot

Loading

tuliren commented Sep 26, 2022 •

edited by github-actions bot

Loading

tuliren commented Sep 26, 2022 •

edited by github-actions bot

Loading

tuliren commented Sep 26, 2022 •

edited by github-actions bot

Loading

tuliren commented Sep 26, 2022

tuliren commented Sep 26, 2022

🎉 New Destination: R2 Cloudflare #15296

🎉 New Destination: R2 Cloudflare #15296

Conversation

videnkz commented Aug 4, 2022 • edited by tuliren Loading

What

🚨 User Impact 🚨

Tests

Pre-merge checks

CLAassistant commented Aug 4, 2022 • edited Loading

marcosmarxm commented Aug 4, 2022

videnkz commented Sep 7, 2022 • edited Loading

videnkz commented Sep 8, 2022

tuliren commented Sep 26, 2022 • edited by github-actions bot Loading

Build Passed

tuliren commented Sep 26, 2022 • edited by github-actions bot Loading

Build Passed

tuliren commented Sep 26, 2022 • edited by github-actions bot Loading

Build Passed

tuliren commented Sep 26, 2022 • edited by github-actions bot Loading

Build Passed

tuliren commented Sep 26, 2022 • edited by github-actions bot Loading

Build Passed

tuliren commented Sep 26, 2022 • edited by github-actions bot Loading

Build Passed

tuliren commented Sep 26, 2022 • edited by github-actions bot Loading

Build Passed

tuliren commented Sep 26, 2022

tuliren commented Sep 26, 2022

videnkz commented Aug 4, 2022 •

edited by tuliren

Loading

CLAassistant commented Aug 4, 2022 •

edited

Loading

videnkz commented Sep 7, 2022 •

edited

Loading

tuliren commented Sep 26, 2022 •

edited by github-actions bot

Loading

tuliren commented Sep 26, 2022 •

edited by github-actions bot

Loading

tuliren commented Sep 26, 2022 •

edited by github-actions bot

Loading

tuliren commented Sep 26, 2022 •

edited by github-actions bot

Loading

tuliren commented Sep 26, 2022 •

edited by github-actions bot

Loading

tuliren commented Sep 26, 2022 •

edited by github-actions bot

Loading

tuliren commented Sep 26, 2022 •

edited by github-actions bot

Loading