-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🎉Redshift, Databricks, Snowflake, S3 Destinations: Make S3 output filename configurable #14494
🎉Redshift, Databricks, Snowflake, S3 Destinations: Make S3 output filename configurable #14494
Conversation
/test connector=connectors/destination-redshift
Build PassedTest summary info:
|
/test connector=connectors/destination-snowflake
Build PassedTest summary info:
|
/test connector=connectors/destination-s3
Build PassedTest summary info:
|
/test connector=connectors/destination-databricks
Build PassedTest summary info:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please do not forget to pass PR's checklist and run checkstyle command:
./gradlew --no-daemon format --scan
Thanks
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
Please run also integration tests for the GCS, BigQuery, and BigQuery Denormalized. They might be affected.
/test connector=connectors/destination-gcs
Build PassedTest summary info:
|
/test connector=connectors/destination-bigquery
Build PassedTest summary info:
|
/test connector=connectors/destination-bigquery-denormalized
Build PassedTest summary info:
|
@DoNotPanicUA Will be done, sir. |
One more integration-test run iteration to test if everything works, after master was merged. |
/test connector=connectors/destination-gcs
Build PassedTest summary info:
|
/test connector=connectors/destination-bigquery
Build PassedTest summary info:
|
/test connector=connectors/destination-bigquery-denormalized
Build PassedTest summary info:
|
/test connector=connectors/destination-s3
Build PassedTest summary info:
|
/test connector=connectors/destination-redshift
Build PassedTest summary info:
|
/test connector=connectors/destination-databricks
Build PassedTest summary info:
|
/test connector=connectors/destination-snowflake
Build PassedTest summary info:
|
/publish connector=connectors/destination-redshift,connectors/destination-snowflake,connectors/destination-s3,connectors/destination-databricks
if you have connectors that successfully published but failed definition generation, follow step 4 here |
…rbytehq-master * 'master' of https://github.com/airbytehq/airbyte: (1141 commits) pass USE_STREAM_CAPABLE_STATE env var to containers/deployments (airbytehq#14737) Bump mqtt connector (airbytehq#14648) Add error code to ManualOperationResult (airbytehq#14657) Bump elasticsearch version (airbytehq#14640) Ryan/sync oracle version number (airbytehq#14736) Fixed linter issue with add_fields.py comments (airbytehq#14742) 🎉Redshift, Databricks, Snowflake, S3 Destinations: Make S3 output filename configurable (airbytehq#14494) 🐛Source-mssql: aligned regular and cdc syncs and its datatype tests (airbytehq#14379) 🎉 Source Amazon Seller Partner: Add new streams (airbytehq#13604) bump source-file-secure (airbytehq#14704) 🎉 New source: Timely airbytehq#13292 (airbytehq#14335) 🪟🔧 Refactor feature service (airbytehq#14559) [low code cdk] add a transformation for adding fields into an outgoing record (airbytehq#14638) Bump destination-postgres to 0.3.21 (airbytehq#14479) Remove `additionalProperties: false` from JDBC destination connectors (airbytehq#14618) 🎉 Source Notion: add OAuth authorization for source-notion connector (airbytehq#14706) Use the configuration diff calculation in the update endpoint (airbytehq#14626) 🪟 🐛 Fix input validation on blur and cleanup signup error handling (airbytehq#14724) lower sleep after wait for successful job (airbytehq#14725) Add configuration diff (airbytehq#14603) ...
What
In #4184 was suggested to add an input to make S3 stage filename configurable.
How
Added
S3FilenameTemplateManager.class
to support this feature. Please, see the code.Recommended reading order
x.java
🚨 User Impact 🚨
There is no impact, if OLD specification will be used without file_name_pattern key, default naming convention will be used.
Pre-merge Checklist
Expand the relevant checklist and delete the others.
New Connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampledocs/integrations/README.md
airbyte-integrations/builds.md
Airbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing/publish
command described hereUpdating a connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampleAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing/publish
command described hereConnector Generator
-scaffold
in their name) have been updated with the latest scaffold by running./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates
then checking in your changesTests
Unit
Put your unit tests output here.
Integration
Put your integration tests output here.
Acceptance
Put your acceptance tests output here.