Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🎉Redshift, Databricks, Snowflake, S3 Destinations: Make S3 output filename configurable #14494

Merged
merged 19 commits into from
Jul 15, 2022

Conversation

alexandertsukanov
Copy link
Contributor

What

In #4184 was suggested to add an input to make S3 stage filename configurable.

How

Added S3FilenameTemplateManager.class to support this feature. Please, see the code.

Recommended reading order

  1. x.java

🚨 User Impact 🚨

There is no impact, if OLD specification will be used without file_name_pattern key, default naming convention will be used.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

  • Community member? Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
    • docs/integrations/README.md
    • airbyte-integrations/builds.md
  • PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • If new credentials are required for use in CI, add them to GSM. Instructions.
  • /test connector=connectors/<name> command is passing
  • New Connector version released on Dockerhub by running the /publish command described here
  • After the connector is published, connector added to connector index as described here
  • Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here
Updating a connector

Community member or Airbyter

  • Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • If new credentials are required for use in CI, add them to GSM. Instructions.
  • /test connector=connectors/<name> command is passing
  • New Connector version released on Dockerhub and connector version bumped by running the /publish command described here
Connector Generator
  • Issue acceptance criteria met
  • PR name follows PR naming conventions
  • If adding a new generator, add it to the list of scaffold modules being tested
  • The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
  • Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

@alexandertsukanov
Copy link
Contributor Author

alexandertsukanov commented Jul 7, 2022

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2629989448
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2629989448
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         162     34    79%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 570    377    34%
-------------------------------------------------------------------------------------
TOTAL                                                              1363    587    57%

Build Passed

Test summary info:

All Passed

@alexandertsukanov
Copy link
Contributor Author

alexandertsukanov commented Jul 7, 2022

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2629992373
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2629992373
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         162     34    79%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 570    377    34%
-------------------------------------------------------------------------------------
TOTAL                                                              1363    587    57%

Build Passed

Test summary info:

All Passed

@alexandertsukanov
Copy link
Contributor Author

alexandertsukanov commented Jul 7, 2022

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2630286098
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2630286098
No Python unittests run

Build Passed

Test summary info:

All Passed

@alexandertsukanov
Copy link
Contributor Author

alexandertsukanov commented Jul 7, 2022

/test connector=connectors/destination-databricks

🕑 connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/2630641551
✅ connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/2630641551
No Python unittests run

Build Passed

Test summary info:

All Passed

Copy link
Contributor

@etsybaev etsybaev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not forget to pass PR's checklist and run checkstyle command:
./gradlew --no-daemon format --scan

Thanks

Copy link
Contributor

@DoNotPanicUA DoNotPanicUA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍
Please run also integration tests for the GCS, BigQuery, and BigQuery Denormalized. They might be affected.

@alexandertsukanov
Copy link
Contributor Author

alexandertsukanov commented Jul 11, 2022

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2650339343
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2650339343
No Python unittests run

Build Passed

Test summary info:

All Passed

@alexandertsukanov
Copy link
Contributor Author

alexandertsukanov commented Jul 11, 2022

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2650339849
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2650339849
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         162     34    79%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 570    377    34%
-------------------------------------------------------------------------------------
TOTAL                                                              1363    587    57%

Build Passed

Test summary info:

All Passed

@alexandertsukanov
Copy link
Contributor Author

alexandertsukanov commented Jul 11, 2022

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2650342095
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2650342095
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         162     34    79%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 570    377    34%
-------------------------------------------------------------------------------------
TOTAL                                                              1363    587    57%

Build Passed

Test summary info:

All Passed

@alexandertsukanov
Copy link
Contributor Author

👍 Please run also integration tests for the GCS, BigQuery, and BigQuery Denormalized. They might be affected.

@DoNotPanicUA Will be done, sir.

@alexandertsukanov alexandertsukanov marked this pull request as ready for review July 12, 2022 08:57
@alexandertsukanov alexandertsukanov requested a review from a team as a code owner July 12, 2022 08:57
@alexandertsukanov
Copy link
Contributor Author

One more integration-test run iteration to test if everything works, after master was merged.

@alexandertsukanov
Copy link
Contributor Author

alexandertsukanov commented Jul 15, 2022

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2675500337
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2675500337
No Python unittests run

Build Passed

Test summary info:

All Passed

@alexandertsukanov
Copy link
Contributor Author

alexandertsukanov commented Jul 15, 2022

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2675505086
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/2675505086
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         162     34    79%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 570    377    34%
-------------------------------------------------------------------------------------
TOTAL                                                              1363    587    57%

Build Passed

Test summary info:

All Passed

@alexandertsukanov
Copy link
Contributor Author

alexandertsukanov commented Jul 15, 2022

/test connector=connectors/destination-bigquery-denormalized

🕑 connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2675508107
✅ connectors/destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/2675508107
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         162     34    79%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 570    377    34%
-------------------------------------------------------------------------------------
TOTAL                                                              1363    587    57%

Build Passed

Test summary info:

All Passed

@alexandertsukanov
Copy link
Contributor Author

alexandertsukanov commented Jul 15, 2022

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2675615888
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2675615888
No Python unittests run

Build Passed

Test summary info:

All Passed

@alexandertsukanov
Copy link
Contributor Author

alexandertsukanov commented Jul 15, 2022

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2675718235
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2675718235
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         162     34    79%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 570    377    34%
-------------------------------------------------------------------------------------
TOTAL                                                              1363    587    57%

Build Passed

Test summary info:

All Passed

@alexandertsukanov
Copy link
Contributor Author

alexandertsukanov commented Jul 15, 2022

/test connector=connectors/destination-databricks

🕑 connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/2675782029
✅ connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/2675782029
No Python unittests run

Build Passed

Test summary info:

All Passed

@alexandertsukanov
Copy link
Contributor Author

alexandertsukanov commented Jul 15, 2022

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2675825506
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2675825506
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         162     34    79%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 570    377    34%
-------------------------------------------------------------------------------------
TOTAL                                                              1363    587    57%

Build Passed

Test summary info:

All Passed

@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label Jul 15, 2022
@alexandertsukanov
Copy link
Contributor Author

alexandertsukanov commented Jul 15, 2022

/publish connector=connectors/destination-redshift,connectors/destination-snowflake,connectors/destination-s3,connectors/destination-databricks

🕑 Publishing the following connectors:
connectors/destination-redshift,connectors/destination-snowflake,connectors/destination-s3,connectors/destination-databricks
https://github.com/airbytehq/airbyte/actions/runs/2676372375


Connector Did it publish? Were definitions generated?
connectors/destination-redshift
connectors/destination-snowflake
connectors/destination-s3
connectors/destination-databricks

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets July 15, 2022 11:36 Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets July 15, 2022 12:34 Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets July 15, 2022 12:45 Inactive
@alexandertsukanov alexandertsukanov merged commit 82297a9 into master Jul 15, 2022
@alexandertsukanov alexandertsukanov deleted the otsukanov/airbyte-4184_configurable_filename branch July 15, 2022 12:56
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets July 15, 2022 12:56 Inactive
jsrcodes added a commit to jsrcodes/airbyte that referenced this pull request Jul 15, 2022
…rbytehq-master

* 'master' of https://github.com/airbytehq/airbyte: (1141 commits)
  pass USE_STREAM_CAPABLE_STATE env var to containers/deployments (airbytehq#14737)
  Bump mqtt connector (airbytehq#14648)
  Add error code to ManualOperationResult (airbytehq#14657)
  Bump elasticsearch version (airbytehq#14640)
  Ryan/sync oracle version number (airbytehq#14736)
  Fixed linter issue with add_fields.py comments (airbytehq#14742)
  🎉Redshift, Databricks, Snowflake, S3 Destinations: Make S3 output filename configurable (airbytehq#14494)
  🐛Source-mssql: aligned regular and cdc syncs and its datatype tests (airbytehq#14379)
  🎉 Source Amazon Seller Partner: Add new streams (airbytehq#13604)
  bump source-file-secure (airbytehq#14704)
  🎉 New source: Timely airbytehq#13292 (airbytehq#14335)
  🪟🔧 Refactor feature service (airbytehq#14559)
  [low code cdk] add a transformation for adding fields into an outgoing record (airbytehq#14638)
  Bump destination-postgres to 0.3.21 (airbytehq#14479)
  Remove `additionalProperties: false` from JDBC destination connectors (airbytehq#14618)
  🎉 Source Notion: add OAuth authorization for source-notion connector (airbytehq#14706)
  Use the configuration diff calculation in the update endpoint (airbytehq#14626)
  🪟 🐛 Fix input validation on blur and cleanup signup error handling (airbytehq#14724)
  lower sleep after wait for successful job (airbytehq#14725)
  Add configuration diff (airbytehq#14603)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make S3 output filename configurable
6 participants